Welcome to the World of Big Data Analytics

Transkrypt

1

2 Welcome to the World of Big Data Analytics Grzegorz Jancik Business Development Manager B3System S.A. Franck Sidi EMEA Technology Architect Data Computing Division EMC Corporation 1

3 Kompetencje Wydajna i niezawodna Infrastruktura IT w warstwie sieci, serwerów i macierzy dyskowych, systemów operacyjnych oraz aplikacji realizowana w oparciu o najnowocześniejsze technologie i uznane standardy Kompleksowe bezpieczeństwo IT w obszarach tworzenia i audytu polityk bezpieczeństwa, zabezpieczania przed dostępem do danych i przed wyciekiem informacji, systemów antyfraudowych oraz bezpieczeństwa sieci Zarządzanie usługami i systemami IT w zakresie monitoringu operacyjnego, zarządzania konfiguracją, analizy wydajnościowej aplikacji i systemów klasy help desk realizowane w oparciu o szeroki zbiór praktyk ITIL Cloud computing - złożone projekty polegające na budowie wysokodostępnej i elastycznej infrastruktury informatycznej służącej jako platforma do świadczenia usług informatycznych na rzecz klientów wewnętrznych (private cloud) jak i zewnętrznych (public cloud)

4 Kompetencje w zakresie Greenplum Zespół przeszkolonych specjalistów w obszarze EMC Greenplum Doświadczenie we wdrażaniu rozwiązań opartych o Oracle/MS SQL Specjaliści z zakresu BI, analitycy, programiści Wysokie kompetencje w zakresie infrastruktury (HA, Backup,) 3

5 Dlaczego EMC Greenplum? Innowacyjna technologia Doskonałe TCO Unikalna na rynku wydajność rozwiązania Łatwa integracja z istniejącym środowiskiem Rozwiązanie dostępne jako HW lub SW Kompleksowa platforma dla wielkich wolumenów danych 4

6 Greenplum: EMC s Data Computing Division First Product Shipped Founded in 2003, HQ in San Mateo - USA Reached 100 Customers Reached 300 Customers Software for Large Scale Data Warehouse Installations from small (TBs) to very large (PBs) Acquired by EMC (29 th July 2010) became the foundation of EMC s Data Computing Division (DCD) 150 People in January People December 2011 (R&D multiply by 3)

7 We Help Manage a lot of Data 30 Trillion Rows of data being managed by Greenplum databases for 300 Customers

8 Our Customers Include 300+ global customers 120+ Customers using Greenplum Appliances $250+ Million saved by customers choosing Greenplum over Teradata 5+ Billion shares analyzed daily by Financial Markets using Greenplum 20+ Trillion rows being mined for business value 1+ Billion consumers and constituents receiving more secure and personalized services from Greenplum customers

9 Our Customers Tackle Interesting Problems Advertising and Communications Improve Ad Targeting through deep analysis of Big data. Real-time Analysis of Marketing Campaign. Move from Oracle. 28TB Database Size Retail (Walmart.com) Direct marketing/crm Financial Services Detect and prevent fraud and credit scoring and analysis to reduce credit risk Financial Markets Analysis of all trade and transaction data on NASDAQ. 30 Times Faster compare to previous system. Adding 1 to 2TB of Data Per Day Internet 1 trillion row fact table, adding 3TB/day Clickstream analytics for ad targeting and market research

10 Overall Partner Network and growing Hardware Vendors BI / ETL Tools Solutions & OEM Consultants And Resellers

11 Information Flow in a DW/BI Environment Data Sources CRM Business Analysis BI Tools ERP Supply Chain Web Extract, Transform, Load (ETL) Data Warehouse (RDBMS) Data Mining Users Financial Systems Query Reports Call Center Data Input ETL Data Warehouse Data Analysis Presentation & Delivery

12 Databases Need to Adapt to Big Data Traditional RDBMS is not optimized for Big Data Analytics 50% of TDWI survey respondents will replace their DW platform in the next 3 years because: Cannot do advanced analysis Cannot handle big data volumes Poor query response Can t support advanced analytics Inadequate data load speed Can t scale up to large date volumes Cost of scaling up is too expensive Poorly suited to real-time or on-demand workloads 40% 39% 37% 33% 29% 45% Source: TDWI Next Gen Database Study, 2010

13 What is Greenplum Database? EMC Greenplum is a SQL Database, using a shared nothing, massively parallel processing (MPP) Architecture for Data Warehouse solutions Can Load Equivalent of Population of Poland Every 5 Second 8M Rows / Second in 8 Commodity Servers Configuration Fast Data Loading Extreme Query Performance & Linear Scalability on Commodity HW

14 Architecture of Greenplum SQL MapReduce R,Java,C,Python,Perl ODBC, JDBC, OLEDB BI/ETL Tools... Master Master Greenplum MPP (Massively Parallel Processing) - Shared Nothing Extreme Scale based on Commodity Hardware X86 Fault Tolerant Architecture Automated Parallelization & Tuning Segment Segment Segment Segment Segment Just Load And Query, No Index External Files, URLs, Hadoop (HDFS), WebServices (including from other DBs),

15 Linear Scalability - Predictable Note that the shared-nothing architecture that Greenplum is built upon is scalable So we could double the number of nodes and halve the scan time Since we scale the compute and the I/O together We could double the number of nodes and double the amount of data and keep the scan time constant Commodity Hardware X86 Segment Segment Segment Segment Segment

16 EMC Greenplum Key Use Cases 1. Data loading issues 2. Query performance problems 3. Data warehouse capacity growth 4. Data mart consolidation 5. Advanced analytics / reporting The power to say YES

17 Parallel In-Database Reporting and Advanced Analytics In-database Analytics Bringing the power of parallelism MapReduce Big data predictive analytics with any tool PL/R & Tools Integration R functions can be written directly in the database MAD library A library of advanced analytics functions Recent examples include logistic regression, the Mann-Whitney U test, the Chi Square test, K-means clustering,

18 A Strategic Partnership for High-Performance Computing and MAD Analytics Access relational data-sets for agile analysis SAS/ACCESS provides fast, transparent and secure access to Greenplum data. Leverage database scalability for rapid model deployment SAS Scoring Accelerator publishes models for execution in parallel across the Greenplum cluster. Build complex models at massive scales The SAS/Greenplum Analytics Appliance combines SAS In- Memory Analytics with Greenplum parallelism to produce record-breaking scalability and performance.

19 Big Data Analytical Applications Gene Sequencing Unstructured Data Movie Editing Seismic Study Semi-Structured Data Social Media Clickstream Productivity Telco Billing Structured Data Retail POS Sales Forecast Web Content Storage Services Social Media Clickstream Productivity Hybrid Cloud

20 Greenplum and Hadoop Analytics Structured ERP/CRM Semi-Structured Machine Data Logs UnStructured Images/Sound Ad-hoc Analysis Dynamic Data batch reporting on static data

21 Greenplum Data Co-Processing with Hadoop Greenplum Database Analytic Productivity Applications, Tools, Chorus Data Computing Interfaces SQL, MapReduce, In-Database Analytics, Parallel Data Loading (batch or real-time) Hadoop Compute Storage parallel data exchange Compute Storage SQL DB Engine parallel data exchange MapReduce Engine Network unstructured data structured data temporal data All Data Types geospatial data sensor data spatial data

22 Greenplum Data Computing Appliances Application Specific Configurations DATABASE HADOOP Purpose-built, Highly Scalable Data Warehousing Appliance That Delivers Leading Price Performance Greenplum Database Combined With SAS High-performance Computing To Enable Analytics On All The Data Greenplum Database Combined With Hadoop To Enable Co-processing Of Structured And Unstructured Data EMC* makes no representation and undertakes no obligations with regard to product planning information, anticipated product characteristics, performance specifications, or anticipated release dates (collectively, Roadmap Information ). Roadmap Information is provided by EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound thereby.

23 How to Migrate to Greenplum Greenplum & B3System are providing Tools & Methodologies to accelerate migration to Greenplum Meeting to discuss Global Architecture and scope of the migration Send a list of questions per domain Conduct Workshop (1-2h Meeting) per Domain Conduct Assessment Define Statement of Work for the Migration Start Project

24 How to Automate Migration Migration of SQL Code SQLWays Migration of Data WisdomForce Informatica

25 Greenplum Delivers Choice & Flexibility Data Computing Appliance Choose Greenplum Database And/Or Hadoop Modules In ¼ Rack Increments Scale Up By Adding Your Choice Of Additional Modules Minimal Time To Value Greenplum Software Solutions Greenplum Database, Hadoop, & Chorus On Your X86 Hardware Flexibility For Any Workload Or Environment Perpetual Or Subscription Licenses Price per TB (Whatever the number of processors or cores or servers)

26 Greenplum Delivers Complete Big Data Analytics platform 10x performance Load improvement Ability to run query-intensive reports in seconds, not hours Simple integration with existing infrastructure, tools and methodologies The power to say YES

27 THANK YOU Q&A

28 Grzegorz Jancik B3System S.A. Franck Sidi EMC Data Computing Products Division B3System S.A. Aleje Jerozolimskie 162 A, II pietro Warszawa tel. +48 (22) fax. +48 (22)