Back To Schedule
Thursday, September 24 • 9:30am - 10:20am
Instantly Finding a Needle of Data in a Haystack of Large-Scale NFS Environment

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Intel Design environment heavily depends on a large scale NFS infrastructure with 10s of PBs of data. Global Name space helps to navigate this large environment in a uniform way from 60,000 compute servers.

But what if a user doesn't know where the piece of data he is looking for is located?

Our customers used to spend hours waiting for recursive ""grep"" commands' completion - or preferred not to bother with some less critical queries.

In this talk, we'll cover how Intel IT has identified an opportunity to provide a faster way to look for an information within this large-scale NFS environment. We'll review various open source solutions which were considered, and how we've decided to implement a mix of home-grown scalable NFS crawler with open source ElasticSearch engine to index parts of our NFS environment.

As part of this talk we'll discuss various challenges and our ways to mitigate them, including:

crawler scalability required to index large amounts of dynamically changing data within pre-defined indexing SLA
Index scalability and performance requirements
Relevancy of the results presented in search queries by customers
User interface considerations
Security aspects of the index access control
This might be an interesting conversation for both storage vendors - covering a useful feature which might be implemented as a part of NFS environment, and for storage customers who may benefit from such capability.

Learning Objectives

How to implement scalable indexing and search on top of large scale NFS
Scalable crawling with controlled performance impact on shared file servers
Security aspects of data index and search representation

avatar for Gregory Touretsky

Gregory Touretsky

Product Manager, Infinidat
Gregory Touretsky has recently joined Infinidat as a Product Manager. Prior to that he has been Solutions Architect within Intel IT, focusing on distributed computing and storage solutions, data sharing and cloud. Gregory holds MSc in Computers Engineering from Novosibirsk State Technical... Read More →

Thursday September 24, 2015 9:30am - 10:20am PDT
Lafayette Room

Attendees (0)