Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-8448b6f56d-c47g7 Total loading time: 0 Render date: 2024-04-25T03:59:51.182Z Has data issue: false hasContentIssue false

12 - Applications

from Part V - Case studies

Published online by Cambridge University Press:  05 March 2014

Henrique C. M. Andrade ,
Buğra Gedik  and
Deepak S. Turaga
Henrique C. M. Andrade
Affiliation:
J. P. Morgan
Buğra Gedik
Affiliation:
Bilkent University, Ankara
Deepak S. Turaga
Affiliation:
IBM Thomas J. Watson Research Center, New York
Get access

Summary

Overview

In previous chapters we covered different aspects of the stream processing paradigm, including its software engineering and systems aspects, as well as its analytical foundations. In this chapter, we make use of this knowledge by taking a closer look into the design of real-world applications.

We present three applications from different business domains: a large-scale systems operations monitoring application used to manage a portion of an IT software environment, a patient monitoring application used in the context of clinical healthcare, and an application for semiconductor fabrication process control used in a manufacturing environment.

Each of these applications is described from the perspective of a system analyst who has to design the appropriate software solution. Hence, we cover the entire software engineering process, starting from gathering the requirements, designing and implementing the application components, and finalizing with the stages of optimization and refinement.

The first application, which we label Operations Monitoring (OM), is focused on analyzing log messages and events capturing the resource usage of various operational processes in a large-scale networked system shared by multiple users and processes (Section 12.2). Its aim is to continuously monitor the state of that environment, enforcing appropriate resource usage policies, as well as detecting (and potentially predicting) system outages.

This application is a front-end to different analytic applications. Its primary challenge consists of dealing with the scalability and performance requirements associated with handling large volumes of streaming data.

Type
Chapter
Information
Fundamentals of Stream Processing
Application Design, Systems, and Analytics
 , pp. 441 - 484
Publisher: Cambridge University Press
Print publication year: 2014

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

[1] Weka Data Mining in Java; retrieved in December 2010. http://www.cs.waikato.ac.nz/ml/weka/.
[2] Kopparapu, C. Load Balancing Servers, Firewalls, and Caches. John Wiley & Sons, Inc.; 2002.Google Scholar
[3] Streams Exchange; retrieved in May 2012. https://www.ibm.com/developer works/mydeveloperworks/groups/service/html/communityview?communityUuid=d4e7dc8d-0efb-44ff-9a82-897202a3021e.
[4] Griffin, MP, Moorman, JR. Toward the early diagnosis of neonatal sepsis and sepsis-like illness using novel heart rate analysis. Pediatrics. 2001;107(1):97–104.CrossRefGoogle ScholarPubMed
[5] Vespa, P, Nuwer, M, Juhasz, C, Alexander, M, Nenov, V, Martin, N, et al.Early detection of vasospasm after acute subarachnoid hemorrhage using continuous EEG ICU monitoring. Electroencephalography and Clinical Neurophysiology. 1997;103(6):607–615.CrossRefGoogle ScholarPubMed
[6] Blount, M, Ebling, M, Eklund, M, James, A, McGregor, C, Percival, N, et al.Real-time analysis for intensive care: development and deployment of the Artemis analytic system. IEEE Engineering in Medicine and Biology Magazine. 2010;29(2):110–118.CrossRefGoogle ScholarPubMed
[7] Bonow, R, Mann, D, Zipes, D, Libby, P. Braunwald's Heart Disease: A Textbook of Cardiovascular Medicine. Elsevier Saunders Co.; 2012.Google Scholar
[8] Pan, J, Tompkins, WJ. A real-time QRS detection algorithm. IEEE Transactions on Biomedical Engineering (TBME). 1985;32(3):230–236.Google ScholarPubMed
[9] Hamilton, PS, Tompkins, WJ. Quantitative investigation of QRS detection rules using the MIT/BIH arrhythmia database. IEEE Transactions on Biomedical Engineering (TBME). 1986;33(12):1157–1165.Google ScholarPubMed
[10] CapsuleTech; retrieved in December 2010. http://www.capsuletech.com/.
[11] Electronic Industries Association – Engineering Department. Interface Between Data Terminal Equipment and Data Communication Equipment Employing Serial Binary Data Interchange. EIA; 1969.
[12] MIT PhysioBank: Arrhythmia Database; retrieved in December 2010. http://www.physionet.org/physiobank/database/mitdb/.
[13] Hastie, T, Tibshirani, R, Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer; 2009.CrossRefGoogle Scholar
[14] Vieira Neto, H, Nehmzow, U. Incremental PCA: an alternative approach for novelty detection. In: Proceedings of Autonomous Robotic Systems Conference (TAROS). London, UK; 2005. pp. 227–233.Google Scholar
[15] Duda, RO, Hart, PE. Pattern Classification and Scene Analysis. John Wiley & Sons, Inc.; 1973.Google Scholar
[16] IEEE. Standard for a smart transducer interface for sensors and actuators. IEEE Standards Association; 2007. IEEE 1451.5-2007.
[17] Gardner, WA. Learning characteristics of stochastic-gradient-descent algorithms: A general study, analysis, and critique. Signal Processing. 1984;6(2):113–133.CrossRefGoogle Scholar
[18] Hall, P, Park, BU, Samworth, RJ. Choice of neighbor order in nearest-neighbor classification. Annals of Statistics. 2008;36(5):2135–2152.CrossRefGoogle Scholar
[19] Aggarwal, CC. A framework for diagnosing changes in evolving data streams. In: Proceedings of the ACM International Conference on Management of Data (SIGMOD). San Diego, CA; 2003. pp. 575–586.Google Scholar
[20] Dasu, T, Krishnan, S, Venkatasubramaniam, S, Yi, K. An information-theoretic approach to detecting changes in multi-dimensional data streams. In: Proceedings of the Symposium on the Interface of Statistics, Computing Science, and Applications (Interface). Pasadena, CA; 2006.Google Scholar
[21] Kifer, D, Ben-David, S, Gehrke, J. Detecting change in data streams. In: Proceedings of the International Conference on Very Large Databases (VLDB). Toronto, Canada; 2004. pp. 180–191.Google Scholar
[22] Dongarra, J, Duff, I, Sorensen, D, van der Vorst, H, editors. Numerical Linear Algebra for High Performance Computers. Society for Industrial and Applied Mathematics; 1998.CrossRef
[23] Adobe. Using Adobe Flex 4.6. Adobe Systems; 2011.
[24] Apache Flex; retrieved in September 2012. http://http://incubator.apache.org/flex/.

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×