Implementing Splunk: Big Data Essentials for Operational Intelligence ppt

448 2.6K 1
Implementing Splunk: Big Data Essentials for Operational Intelligence ppt

Đang tải... (xem toàn văn)

Tài liệu hạn chế xem trước, để xem đầy đủ mời bạn chọn Tải xuống

Thông tin tài liệu

www.it-ebooks.info Implementing Splunk: Big Data Reporting and Development for Operational Intelligence Learn to transform your machine data into valuable IT and business insights with this comprehensive and practical tutorial Vincent Bumgarner BIRMINGHAM - MUMBAI www.it-ebooks.info Implementing Splunk: Big Data Reporting and Development for Operational Intelligence Copyright © 2013 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: January 2013 Production Reference: 1140113 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-84969-328-8 www.packtpub.com Cover Image by Vincent Bumgarner (vincent.bumgarner@gmail.com) www.it-ebooks.info Credits Author Vincent Bumgarner Reviewers Mathieu Dessus Cindy McCririe Nick Mealy Acquisition Editor Kartikey Pandey Lead Technical Editor Azharuddin Sheikh Technical Editors Charmaine Pereira Varun Pius Rodrigues Copy Editors Brandt D'Mello Aditya Nair Alda Paiva Laxmi Subramanian Ruta Waghmare Project Coordinator Anish Ramchandani Proofreader Martin Diver Indexer Tejal Soni Graphics Aditi Gajjar Production Coordinator Nitesh Thakur Cover Work Nitesh Thakur www.it-ebooks.info About the Author Vincent Bumgarner has been designing software for nearly 20 years, working in many languages on nearly as many platforms. He started using Splunk in 2007 and has enjoyed watching the product evolve over the years. While working for Splunk, he helped many companies, training dozens of users to drive, extend, and administer this extremely exible product. At least one person at every company he worked with asked for a book on Splunk, and he hopes his effort helps ll their shelves. I would like to thank my wife and kids as this book could not have happened without their support. A big thank you to all of the reviewers for contributing their time and expertise, and special thanks to SplunkNinja for the recommendation. www.it-ebooks.info About the Reviewers Mathieu Dessus is a security consultant for Verizon in France and acts as the SIEM leader for EMEA. With more than 12 years of experience in the security area, he has acquired a deep technical background in the management, design, assessment, and systems integration of information security technologies. He specializes in web security, Unix, SIEM, and security architecture design. Cindy McCririe is a client architect at Splunk. In this role, she has worked with several of Splunk's enterprise customers, ensuring successful deployment of the technology. Many of these customers are using Splunk in unique ways. Sample use cases include PCI compliance, security, operations management, business intelligence, Dev/Ops, and transaction proling. Nick Mealy was an early employee at Splunk and worked as the Mad Scientist / Principal User Interface Developer at Splunk from March 2005 to September 2010. He led the technical design and development of the systems that power Splunk's search and reporting interfaces as well as on the general systems that power Splunk's congurable views and dashboards. In 2010, he left Splunk to found his current company, Sideview, which is creating new Splunk apps and new products on top of the Splunk platform. The most widely known of these products is the Sideview Utils app, which has become very widely deployed (and will be discussed in Chapter 8, Building Advanced Dashboards). Sideview Utils provides new UI modules and new techniques that make it easier for Splunk app developers and dashboard creators to create and maintain their custom views and dashboards. www.it-ebooks.info www.PacktPub.com Support les, eBooks, discount offers and more You might want to visit www.PacktPub.com for support les and downloads related to your book. Did you know that Packt offers eBook versions of every book published, with PDF and ePub les available? You can upgrade to the eBook version at www.PacktPub. com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. http://PacktLib.PacktPub.com Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books. Why Subscribe? • Fully searchable across every book published by Packt • Copy and paste, print and bookmark content • On demand and accessible via web browser Free Access for Packt account holders If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access. www.it-ebooks.info Table of Contents Preface 1 Chapter 1: The Splunk Interface 7 Logging in to Splunk 7 The Home app 8 The top bar 11 Search app 13 Data generator 13 The Summary view 14 Search 16 Actions 17 Timeline 18 The eld picker 19 Fields 19 Search results 21 Options 22 Events viewer 23 Using the time picker 25 Using the eld picker 26 Using Manager 27 Summary 29 Chapter 2: Understanding Search 31 Using search terms effectively 31 Boolean and grouping operators 32 Clicking to modify your search 34 Event segmentation 34 Field widgets 34 Time 35 www.it-ebooks.info Table of Contents [ ii ] Using elds to search 35 Using the eld picker 35 Using wildcards efciently 36 Only trailing wildcards are efcient 36 Wildcards are tested last 36 Supplementing wildcards in elds 37 All about time 37 How Splunk parses time 37 How Splunk stores time 37 How Splunk displays time 38 How time zones are determined and why it matters 38 Different ways to search against time 39 Specifying time in-line in your search 41 _indextime versus _time 42 Making searches faster 42 Sharing results with others 43 Saving searches for reuse 46 Creating alerts from searches 48 Schedule 49 Actions 51 Summary 52 Chapter 3: Tables, Charts, and Fields 53 About the pipe symbol 53 Using top to show common eld values 54 Controlling the output of top 56 Using stats to aggregate values 57 Using chart to turn data 61 Using timechart to show values over time 63 timechart options 65 Working with elds 66 A regular expression primer 66 Commands that create elds 68 eval 68 rex 69 Extracting loglevel 70 Using the Extract Fields interface 70 Using rex to prototype a eld 73 Using the admin interface to build a eld 75 Indexed elds versus extracted elds 77 Summary 80 www.it-ebooks.info Table of Contents [ iii ] Chapter 4: Simple XML Dashboards 81 The purpose of dashboards 81 Using wizards to build dashboards 82 Scheduling the generation of dashboards 91 Editing the XML directly 91 UI Examples app 92 Building forms 92 Creating a form from a dashboard 92 Driving multiple panels from one form 97 Post-processing search results 104 Post-processing limitations 106 Panel 1 106 Panel 2 107 Panel 3 108 Final XML 108 Summary 110 Chapter 5: Advanced Search Examples 111 Using subsearches to nd loosely related events 111 Subsearch 111 Subsearch caveats 112 Nested subsearches 113 Using transaction 114 Using transaction to determine the session length 115 Calculating the aggregate of transaction statistics 117 Combining subsearches with transaction 118 Determining concurrency 122 Using transaction with concurrency 122 Using concurrency to estimate server load 123 Calculating concurrency with a by clause 124 Calculating events per slice of time 129 Using timechart 129 Calculating average requests per minute 131 Calculating average events per minute, per hour 132 Rebuilding top 134 Summary 141 Chapter 6: Extending Search 143 Using tags to simplify search 143 Using event types to categorize results 146 Using lookups to enrich data 150 Dening a lookup table le 150 www.it-ebooks.info [...]... Splunk instance types Splunk forwarders Splunk indexer Splunk search Common data sources Monitoring logs on servers Monitoring logs on a shared drive Consuming logs in batch Receiving syslog events Receiving events directly on the Splunk indexer Using a native syslog receiver Receiving syslog with a Splunk forwarder Consuming logs from a database Using scripts to gather data Sizing indexers Planning... Writing commands When not to write a command When to write a command Configuring commands Adding fields Manipulating data [ viii ] www.it-ebooks.info 379 379 380 382 384 385 387 390 390 392 392 393 394 Table of Contents Transforming data Generating data Writing a scripted lookup to enrich data Writing an event renderer Using specific fields Table of fields based on field value Pretty print XML Writing... extend Splunk for data input, external querying, rendering, custom commands, and custom actions What you need for this book To work through the examples in this book, you will need an installation of Splunk, preferably a non-production instance If you are already working with Splunk, then the concepts introduced by the examples should be applicable to your own data Splunk can be downloaded for free from... try to act as a replacement for the official Splunk documentation, but should serve as a shortcut for many concepts For some sections, a good understanding of regular expressions would be helpful For some sections, the ability to read Python would be helpful Conventions In this book, you will find a number of styles of text that distinguish between different kinds of information Here are some examples... to Splunk We will discuss apps in a more detailed manner in Chapter 7, Working with Apps Under Do more with Splunk, we find: • Add data: This links to the Add Data to Splunk page This interface is a great start for getting local data flowing into Splunk The new Preview data interface takes an enormous amount of complexity out of configuring dates and line breaking We won't go through those interfaces... For instance, if the system is configured to use Windows Active Directory via LDAP (a very common configuration), users must change their password in Windows Search app The search app is where most actions in Splunk start Data generator If you want to follow the examples that appear in the next few chapters, install the ImplementingSplunkDataGenerator demo app by following these steps: 1 Download ImplementingSplunkDataGenerator.tar.gz... indexed data panel shows statistics for all indexed data Remember that this only reflects indexes that this particular user searches by default There are other events that are indexed by Splunk, including events Splunk indexes about itself We will discuss indexes in Chapter 9, Building Advanced Dashboards The next three panels give a breakdown of your data using three important pieces of metadata—source,... be thousands of machines submitting data, but all data at the same path across these machines counts as one source When the data source is not a file, the value of the source can be arbitrary, for instance the name of a script or network port A source type is an arbitrary categorization of events There may be many sources across many hosts in the same source type For instance, given the sources /var/... which provides very detailed information about the query that was run • Print ( ), which formats the page for printing and instructs the browser to print [ 17 ] www.it-ebooks.info The Splunk Interface • Save, which provides different options for saving the search or the results We will discuss this later in this chapter • Create, which provides wizard-like interfaces for building different objects... http://www.splunk.com/download, for most popular platforms The sample code was developed on a Unix system, so you will probably have better luck using an installation of Splunk that is running on a Unix operating system Knowledge of Python is necessary to follow some of the examples in the later chapters [2] www.it-ebooks.info Preface Who this book is for This book should be useful for new users, seasoned . www.it-ebooks.info Implementing Splunk: Big Data Reporting and Development for Operational Intelligence Learn to transform your machine data into valuable. Bumgarner BIRMINGHAM - MUMBAI www.it-ebooks.info Implementing Splunk: Big Data Reporting and Development for Operational Intelligence Copyright © 2013 Packt Publishing All

Ngày đăng: 07/03/2014, 04:20

Từ khóa liên quan

Mục lục

  • Cover

  • Copyright

  • Credits

  • About the Author

  • About the Reviewers

  • www.PacktPub.com

  • Table of Contents

  • Preface

  • Chapter 1: The Splunk Interface

    • Logging in to Splunk

    • The Home app

    • The top bar

    • Search app

      • Data generator

      • The Summary view

      • Search

      • Actions

      • Timeline

      • The field picker

        • Fields

      • Search results

        • Options

        • Events viewer

    • Using the time picker

    • Using the field picker

    • Using Manager

    • Summary

  • Chapter 2: Understanding Search

    • Using search terms effectively

    • Boolean and grouping operators

    • Clicking to modify your search

      • Event segmentation

      • Field widgets

      • Time

    • Using fields to search

      • Using the field picker

    • Using wildcards efficiently

      • Only trailing wildcards are efficient

      • Wildcards are tested last

      • Supplementing wildcards in fields

    • All about time

      • How Splunk parses time

      • How Splunk stores time

      • How Splunk displays time

      • How time zones are determined and why it matters

      • Different ways to search against time

      • Specifying time in-line in your search

      • _indextime versus _time

    • Making searches faster

    • Sharing results with others

    • Saving searches for reuse

    • Creating alerts from searches

      • Schedule

      • Actions

    • Summary

  • Chapter 3: Tables, Charts, and Fields

    • About the pipe symbol

    • Using top to show common field values

      • Controlling the output of top

    • Using stats to aggregate values

    • Using chart to turn data

    • Using timechart to show values over time

      • timechart options

    • Working with fields

      • A regular expression primer

      • Commands that create fields

        • eval

        • rex

      • Extracting loglevel

        • Using the Extract Fields interface

        • Using rex to prototype a field

        • Using the admin interface to build a field

        • Indexed fields versus extracted fields

    • Summary

  • Chapter 4: Simple XML Dashboards

    • Why build a dashboard?

    • Using wizards to build dashboards

    • Scheduling the generation of dashboards

    • When to edit the XML directly?

    • UI Examples app

    • Building Forms

      • Creating a form from a dashboard

      • Driving multiple panels from one form

      • Post-processing search results

      • Post-processing limitations

        • Panel 1

        • Panel 2

        • Panel 3

        • Final XML

    • Summary

  • Chapter 5: Advanced Search Examples

    • Using subsearches to find loosely related events

      • Subsearch

      • Subsearch caveats

      • Nested subsearches

    • Using transaction

      • Using transaction to determine the session length

      • Calculating the aggregate of transaction statistics

      • Combining subsearches with transaction

    • Determining concurrency

      • Using transaction with concurrency

      • Using concurrency to estimate server load

      • Calculating concurrency with a by clause

    • Calculating events per slice of time

      • Using timechart

      • Calculating average requests per minute

      • Calculating average events per minute, per hour

    • Rebuilding top

    • Summary

  • Chapter 6: Extending Search

    • Using tags to simplify search

    • Using event types to categorize results

    • Using lookups to enrich data

      • Defining a lookup table file

      • Defining a lookup definition

      • Defining an automatic lookup

      • Troubleshooting lookups

    • Using macros to reuse logic

      • Creating a simple macro

      • Creating a macro with arguments

      • Using eval to build a macro

    • Creating workflow actions

      • Running a new search using values from an event

      • Linking to an external site

      • Building a workflow action to show field context

        • Building the context workflow action

        • Building the context macro

    • Using external commands

      • Extracting values from XML

        • xmlkv

        • XPath

      • Using Google to generate results

    • Summary

  • Chapter 7: Working with Apps

    • Defining an app

    • Included apps

    • Installing apps

      • Installing apps from Splunkbase

        • Using Geo Location Lookup Script

        • Using Google Maps

      • Installing apps from a file

    • Building your first app

    • Editing navigation

    • Customizing the appearance of your app

      • Customizing the launcher icon

      • Using custom CSS

      • Using custom HTML

        • Custom HTML in a simple dashboard

        • Using ServerSideInclude in a complex dashboard

    • Object permissions

      • How permissions affect navigation

      • How permissions affect other objects

      • Correcting permission problems

    • App directory structure

    • Adding your app to Splunkbase

      • Preparing your app

        • Confirming sharing settings

        • Cleaning up our directories

      • Packaging your app

      • Uploading your app

    • Summary

  • Chapter 8: Building Advanced Dashboards

    • Reasons for working with advanced XML

    • Reasons for not working with advanced XML

    • Development process

    • Advanced XML structure

    • Converting simple XML to advanced XML

    • Module logic flow

    • Understanding layoutPanel

      • Panel placement

    • Reusing a query

    • Using intentions

      • stringreplace

      • addterm

    • Creating a custom drilldown

      • Building a drilldown to a custom query

      • Building a drilldown to another panel

      • Building a drilldown to multiple panels using HiddenPostProcess

    • Third-party add-ons

      • Google Maps

      • Sideview Utils

        • The Sideview Search module

        • Linking views with Sideview

        • Sideview URLLoader

        • Sideview forms

    • Summary

  • Chapter 9: Summary Indexes and CSV Files

    • Understanding summary indexes

      • Creating a summary index

    • When to use a summary index

    • When to not use a summary index

    • Populating summary indexes with saved searches

    • Using summary index events in a query

    • Using sistats, sitop, and sitimechart

    • How latency affects summary queries

    • How and when to backfill summary data

      • Using fill_summary_index.py to backfill

      • Using collect to produce custom summary indexes

    • Reducing summary index size

      • Using eval and rex to define grouping fields

      • Using a lookup with wildcards

      • Using event types to group results

    • Calculating top for a large time frame

    • Storing raw events in a summary index

    • Using CSV files to store transient data

      • Prep-opulating a dropdown

      • Creating a running calculation for a day

    • Summary

  • Chapter 10: Configuring Splunk

    • Locating Splunk configuration files

    • The structure of a Splunk configuration file

    • Configuration merging logic

      • Merging order

        • Merging order outside of search

        • Merging order when searching

      • Configuration merging logic

        • Configuration merging example 1

        • Configuration merging example 2

        • Configuration merging example 3

        • Configuration merging example 4 (search)

      • Using btool

    • An overview of Splunk .conf files

      • props.conf

        • Common attributes

        • Stanza types

        • Priorities inside a type

        • Attributes with class

      • inputs.conf

        • Common input attributes

        • Files as inputs

        • Network inputs

        • Native Windows inputs

        • Scripts as inputs

      • transforms.conf

        • Creating indexed fields

        • Modifying metadata fields

        • Lookup definitions

        • Using REPORT

        • Chaining transforms

        • Dropping events

      • fields.conf

      • outputs.conf

      • indexes.conf

      • authorize.conf

      • savedsearches.conf

      • times.conf

      • commands.conf

      • web.conf

    • User interface resources

      • Views and navigation

      • Appserver resources

      • Metadata

    • Summary

  • Chapter 11: Advanced Deployments

    • Planning your installation

    • Splunk instance types

      • Splunk forwarders

      • Splunk indexer

      • Splunk search

    • Common data sources

      • Monitoring logs on servers

      • Monitoring logs on a shared drive

      • Consuming logs in batch

      • Receiving syslog events

        • Receiving events directly on the Splunk indexer

        • Using a native syslog receiver

        • Receiving syslog with a Splunk forwarder

      • Consuming logs from a database

      • Using scripts to gather data

    • Sizing indexers

    • Planning redundancy

      • Indexer load balancing

      • Understanding typical outages

    • Working with multiple indexes

      • Directory structure of an index

      • When to create more indexes

        • Testing data

        • Differing longevity

        • Differing permissions

        • Using more indexes to increase performance

      • The lifecycle of a bucket

      • Sizing an index

      • Using volumes to manage multiple indexes

    • Deploying the Splunk binary

      • Deploying from a tar file

      • Deploying using msiexec

      • Adding a base configuration

      • Configuring Splunk to launch at boot

    • Using apps to organize configuration

      • Separate configurations by purpose

    • Configuration distribution

      • Using your own deployment system

      • Using Splunk deployment server

        • Step 1 – Deciding where your deployment server will run

        • Step 2 – Defining your deploymentclient.conf configuration

        • Step 3 – Defining our machine types and locations

        • Step 4 – Normalizing our configurations into apps appropriately

        • Step 5 – Mapping these apps to deployment clients in serverclass.conf

        • Step 6 – Restarting the deployment server

        • Step 7 – Installing deploymentclient.conf

    • Using LDAP for authentication

    • Using Single Sign On

    • Load balancers and Splunk

      • web

      • splunktcp

      • deployment server

    • Multiple search heads

    • Summary

  • Chapter 12: Extending Splunk

    • Writing a scripted input to gather data

      • Capturing script output with no date

      • Capturing script output as a single event

      • Making a long-running scripted input

    • Using Splunk from the command line

    • Querying Splunk via REST

    • Writing commands

      • When not to write a command

      • When to write a command

      • Configuring commands

      • Adding fields

      • Manipulating data

      • Transforming data

      • Generating data

    • Writing a scripted lookup to enrich data

    • Writing an event renderer

      • Using specific fields

      • Table of fields based on field value

      • Pretty print XML

    • Writing a scripted alert action to process results

    • Summary

  • Index

Tài liệu cùng người dùng

  • Đang cập nhật ...

Tài liệu liên quan