{"id":339,"date":"2019-04-30T18:45:37","date_gmt":"2019-04-30T18:45:37","guid":{"rendered":"http:\/\/algotrading101.com\/wiki\/?p=339"},"modified":"2019-05-15T16:18:42","modified_gmt":"2019-05-15T16:18:42","slug":"big-data","status":"publish","type":"post","link":"https:\/\/algotrading101.com\/wiki\/big-data\/","title":{"rendered":"Big Data"},"content":{"rendered":"\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong>Definition<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p class=\"wp-block-paragraph\">Big data is a field that involves analyzing and managing huge amounts of data.<\/p>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong>Description<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p class=\"wp-block-paragraph\">Similar to smaller data sets, the usual aim of big data is to derive insights from large data sets.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There isn&#8217;t a specific size to determine if a data set is big enough to be considered big data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A data set can be considered big data if the organization has difficulty using traditional methods, software and database to manage their data. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Characteristics of Big Data<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Volume<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to the quantity of data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Velocity<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to the speed at which the data is received and needs processing. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Data from real-time sources usually requires much faster management and processing capabilities, especially when the insights from the data need to be extracted quickly.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Variety<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to the type of data. The common types are:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Text<\/li><li>Numbers<\/li><li>Audio<\/li><li>Imagery<\/li><li>Video<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Another way to categorize data is structured vs unstructured data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Structured data is organised and formatted in a way that is easily searchable, processed and analyzed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Unstructured data has no pre-defined organization or format. This makes it harder to search, process and analyze.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Veracity<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to accuracy of the data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Value<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to how much useful insights can be derived from the data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Variability<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to the consistency of the flow of data. The creation of some data peak during certain times, days or months, but slow down during other times.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><u>Complexity<\/u><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This refers to how complex it is to clean, match, link and manage the data. This characteristic is especially important when there are multiple data sources.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Big data in Industries<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Big data is common in the following industries:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Manufacturing<\/li><li>Media<\/li><li>Government<\/li><li>Social Media<\/li><li>Finance<\/li><li>Healthcare<\/li><li>Insurance<\/li><li>Technology<\/li><\/ul>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong>Examples of Big Data in Action<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<ul class=\"wp-block-list\"><li>Millions of surveillance cameras capture videos of the public across the country. Machine learning is then used to identify faces.<\/li><li>Spotify tracks the data of its users. It then analyzes this data to recommend the users music they might like.<\/li><li> Uber generates and uses a huge amount of data regarding drivers, their vehicles, locations, every trip from every vehicle. These data are analyzed to predict the demand, supply, location of the drivers and decide whether to slap on a surcharge.<\/li><\/ul>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong> Links to Complicated Explanations<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<ul class=\"wp-block-list\"><li><a rel=\"noreferrer noopener nofollow\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/en.wikipedia.org\/wiki\/Big_data\" target=\"_blank\">Big Data &#8211; Wikipedia<\/a><\/li><\/ul>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong>Related Terms<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/algotrading101.com\/wiki\/data-science\/\">Data Science<\/a><\/li><li><a href=\"https:\/\/algotrading101.com\/wiki\/machine-learning\/\">Machine Learning<\/a><\/li><li><a href=\"https:\/\/algotrading101.com\/wiki\/algorithmic-trading\/\">Algorithmic Trading<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Definition Big data is a field that involves analyzing and managing huge amounts of data. Description Similar to smaller data sets, the usual aim of big data is to derive insights from large data sets. There isn&#8217;t a specific size to determine if a data set is big enough to be considered big data. A [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_links_to":"","_links_to_target":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-339","post","type-post","status-publish","format-standard","hentry","category-data-science-resources"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Big Data Simplified - AlgoTrading101 Wiki<\/title>\n<meta name=\"description\" content=\"Quantitative Finance - Explain Like I&#039;m Five\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/algotrading101.com\/wiki\/big-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Big Data Simplified - AlgoTrading101 Wiki\" \/>\n<meta property=\"og:description\" content=\"Quantitative Finance - Explain Like I&#039;m Five\" \/>\n<meta property=\"og:url\" content=\"http:\/\/algotrading101.com\/wiki\/big-data\/\" \/>\n<meta property=\"og:site_name\" content=\"AlgoTrading101 Wiki\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-30T18:45:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-05-15T16:18:42+00:00\" \/>\n<meta name=\"author\" content=\"Lucas Liew\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Lucas Liew\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/#article\",\"isPartOf\":{\"@id\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/\"},\"author\":{\"name\":\"Lucas Liew\",\"@id\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/#\\\/schema\\\/person\\\/bf5a3d12a8fa6a18c56e3dafc22b191f\"},\"headline\":\"Big Data\",\"datePublished\":\"2019-04-30T18:45:37+00:00\",\"dateModified\":\"2019-05-15T16:18:42+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/\"},\"wordCount\":392,\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/\",\"url\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/\",\"name\":\"Big Data Simplified - AlgoTrading101 Wiki\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/#website\"},\"datePublished\":\"2019-04-30T18:45:37+00:00\",\"dateModified\":\"2019-05-15T16:18:42+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/#\\\/schema\\\/person\\\/bf5a3d12a8fa6a18c56e3dafc22b191f\"},\"description\":\"Quantitative Finance - Explain Like I'm Five\",\"breadcrumb\":{\"@id\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\\\/\\\/algotrading101.com\\\/wiki\\\/big-data\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Big Data\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/#website\",\"url\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/\",\"name\":\"AlgoTrading101 Wiki\",\"description\":\"Quantitative Finance in 500 Simple Words\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/#\\\/schema\\\/person\\\/bf5a3d12a8fa6a18c56e3dafc22b191f\",\"name\":\"Lucas Liew\",\"url\":\"https:\\\/\\\/algotrading101.com\\\/wiki\\\/author\\\/algotradingadmin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Big Data Simplified - AlgoTrading101 Wiki","description":"Quantitative Finance - Explain Like I'm Five","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/algotrading101.com\/wiki\/big-data\/","og_locale":"en_US","og_type":"article","og_title":"Big Data Simplified - AlgoTrading101 Wiki","og_description":"Quantitative Finance - Explain Like I'm Five","og_url":"http:\/\/algotrading101.com\/wiki\/big-data\/","og_site_name":"AlgoTrading101 Wiki","article_published_time":"2019-04-30T18:45:37+00:00","article_modified_time":"2019-05-15T16:18:42+00:00","author":"Lucas Liew","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Lucas Liew","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/algotrading101.com\/wiki\/big-data\/#article","isPartOf":{"@id":"http:\/\/algotrading101.com\/wiki\/big-data\/"},"author":{"name":"Lucas Liew","@id":"https:\/\/algotrading101.com\/wiki\/#\/schema\/person\/bf5a3d12a8fa6a18c56e3dafc22b191f"},"headline":"Big Data","datePublished":"2019-04-30T18:45:37+00:00","dateModified":"2019-05-15T16:18:42+00:00","mainEntityOfPage":{"@id":"http:\/\/algotrading101.com\/wiki\/big-data\/"},"wordCount":392,"articleSection":["Data Science"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"http:\/\/algotrading101.com\/wiki\/big-data\/","url":"http:\/\/algotrading101.com\/wiki\/big-data\/","name":"Big Data Simplified - AlgoTrading101 Wiki","isPartOf":{"@id":"https:\/\/algotrading101.com\/wiki\/#website"},"datePublished":"2019-04-30T18:45:37+00:00","dateModified":"2019-05-15T16:18:42+00:00","author":{"@id":"https:\/\/algotrading101.com\/wiki\/#\/schema\/person\/bf5a3d12a8fa6a18c56e3dafc22b191f"},"description":"Quantitative Finance - Explain Like I'm Five","breadcrumb":{"@id":"http:\/\/algotrading101.com\/wiki\/big-data\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/algotrading101.com\/wiki\/big-data\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/algotrading101.com\/wiki\/big-data\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/algotrading101.com\/wiki\/"},{"@type":"ListItem","position":2,"name":"Big Data"}]},{"@type":"WebSite","@id":"https:\/\/algotrading101.com\/wiki\/#website","url":"https:\/\/algotrading101.com\/wiki\/","name":"AlgoTrading101 Wiki","description":"Quantitative Finance in 500 Simple Words","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/algotrading101.com\/wiki\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/algotrading101.com\/wiki\/#\/schema\/person\/bf5a3d12a8fa6a18c56e3dafc22b191f","name":"Lucas Liew","url":"https:\/\/algotrading101.com\/wiki\/author\/algotradingadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/posts\/339","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/comments?post=339"}],"version-history":[{"count":10,"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/posts\/339\/revisions"}],"predecessor-version":[{"id":479,"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/posts\/339\/revisions\/479"}],"wp:attachment":[{"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/media?parent=339"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/categories?post=339"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/algotrading101.com\/wiki\/wp-json\/wp\/v2\/tags?post=339"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}