88 Hands-On Microsoft SQL Server 2008 Integration Services Table 3-1 Partial List of System Variables Available in Various Containers System Variable Assigned to Description CancelEvent Package When set to a nonzero value, the task stops running CreationDate Package Date when the package was created CreatorName Package Name of the person who built this package MachineName Package Name of the computer on which the package runs PackageID Package Unique identifier of the package PackageName Package Name of the package StartTime Package Time that the package started to run UserName Package Account of the user who started the package VersionBuild Package Package version VersionComment Package Comments about the package version TaskID Tasks Unique identifier of a task instance TaskName Tasks Name of the task instance TaskTransactionOption Tasks Transaction option the task uses ErrorCode Event handlers Error identifier ErrorDescription Event handlers Description of the error PercentComplete Event handlers Percentage of completed work SourceDescription Event handlers Description of the executable in the event handler that raised the event SourceID Event handlers Unique identifier of the executable in the event handler that raised the event SourceName Event handlers Name of the executable in the event handler that raised the event VariableDescription Event handlers Variable description VariableID Event handlers Unique identifier of the variable Method You will create a simple package consisting of one task to demonstrate how you can use system variables to create a custom log, which is desirable if standard logging doesn’t meet your purposes. You will create a table in the Campaign database to host logging information and will then use the Execute SQL task to read values of system variables for logging to the earlier created table. You will also add a connection manager to connect to the Campaign database. Chapter 3: Nuts and Bolts of the SSIS Workflow 89 Exercise (Adding a Connection Manager) We will start with creation of CustomLog table and creating an SSIS package in this part. 1. Start SQL Server Management Studio and connect to the database engine of your local server. In the Object Explorer, expand the Databases folder and click Campaign Database. You have created the Campaign database in the last chapter, but if you haven’t, you can attach the Campaign database provided with the software for this book. See the Appendix for more details. Click the New Query button located on the Standard toolbar to open a new query pane. Type the following query in this pane to create a table CustomLog in the Campaign database. USE Campaign CREATE TABLE CustomLog( username varchar(50), machinename varchar(50), packagename varchar(50), packageID varchar(50), taskname varchar(50), starttime datetime) Click Execute on the Toolbar to run the query and create a CustomLog table in the Campaign database. Leave SQL Server Management Studio running as you will return later to see the results. 2. Start BI Development Studio. Choose File | New | Project, or press -- to open the New Project window. 3. Choose Integration Services Project from the Templates pane. In the Name field, enter Use of System Variables. Choose the location C:\SSIS\ Projects, as shown in Figure 3-4. Verify that the Create directory for Solution check box is not selected. Click OK to create a new project. 4. In Solution Explorer, right-click package.dtsx under the SSIS packages folder; then choose Rename. Type CustomLog.dtsx and press . You’ll see a message asking whether you want to rename the package object as well. Click Yes. 5. Locate the Toolbox window, by default, positioned on top-left side of the screen. Hover your mouse over the Toolbox tab to open it. From the Toolbox window, drag and drop the Execute SQL task onto the Control Flow Panel. The Execute SQL task will have a crossed red circle on it. Hovering the mouse on this crossed red circle will show the error message, “No connection manager is specified.” 6. Now create a connection manager for this task. Right-click anywhere in the Connection Managers’ tab and choose New OLE DB Connection from the context menu. This will open the Configure OLE DB Connection Manager dialog box. Click New to open a Connection Manager dialog box. You will find that the Native OLE DB\SQL Server Native Client 10.0 has already been added in the Provider drop-down box. Click the down arrow to look at the other available OLE DB providers. Click Cancel to return. Type localhost in the 90 Hands-On Microsoft SQL Server 2008 Integration Services Server Name field. Leave Use Windows Authentication in the Log On To The Server area. In the Connect To A Database area, make sure the “Select or enter a database name” radio button is selected. Click the down arrow and select the Campaign database from the drop-down list, as shown in Figure 3-5. 7. Click Test Connection to test your connection. You will see a message saying “Test connection succeeded.” Click OK three times to return to the Control Flow Panel. Exercise (Configuring Execute SQL Task) Now is the time to configure the Execute SQL Task. As this is a very versatile task and is probably the most commonly used task, you will use this task in several Hands- On exercises in this book to learn about different uses and configurations that can be applied to this task. 8. Right-click the Execute SQL Task and choose Edit from the context menu to open the Execute SQL Task Editor window. In the SQL Statement group, click in the Connection box and then the down arrow that appears in the far-right corner to choose from the available connections. Notice that you can create a new connection from here as well. Choose localhost.Campaign from the list. Figure 3-4 Creating a new SSIS project for Use of System Variables Chapter 3: Nuts and Bolts of the SSIS Workflow 91 9. Click in the SQLStatement field, and then on the ellipsis button to open the Enter SQL Query window. Type in the following query and then click OK. INSERT INTO CustomLog VALUES (?,?,?,?,?,?) This is a parameterized query in which the question marks represent the parameters. You need to be careful about the order of your parameters while defining them, as they will be considered in the order they are added. 10. Go to Parameter Mapping page to define parameters for your query. Click Add. You will see a system variable being added in the dialog box. You need to change it to the one you need. Click the new variable that has just been added. As you click, the box will turn into a drop-down box. Click the down arrow and choose System::UserName from the list. Similarly, choose VARCHAR in the Data Type column and assign a value of 0 (zero) in the Parameter Name field, as shown in Figure 3-6. Figure 3-5 Adding an OLE DB Connection Manager for Campaign database 92 Hands-On Microsoft SQL Server 2008 Integration Services Now add five more variables by clicking Add. Change the Variable Name, Data Type, and Parameter Name values as per the following list. Click OK to close the window once all the parameters have been defined. Variable Name Direction Data Type Parameter Name System::UserName Input VARCHAR 0 System::MachineName Input VARCHAR 1 System::PackageName Input VARCHAR 2 System::PackageID Input VARCHAR 3 System::TaskName Input VARCHAR 4 System::StartTime Input DATE 5 Figure 3-6 Configuring Parameter Mapping page for Execute SQL task Chapter 3: Nuts and Bolts of the SSIS Workflow 93 Exercise (Executing the Package to Populate CustomLog Table) Finally, let us execute the package and populate CustomLog table. 11. Choose Debug | Start Debugging to run the package in the debugging mode. The Execute SQL task will turn yellow and then green. Click the stop debugging button on the debugging toolbar and press -- to save all the files in the project. 12. Switch back to SQL Server Management Studio. In the query window (open a query window if this is not already open), execute the following query: SELECT * FROM CustomLog. You will see a record entered in the CustomLog table with all the details configured for logging. Review In this exercise, you read values of system variables at the point in time when the package was run and loaded those values to a table in SQL Server. SSIS provides extensive logging facilities using log providers (discussed in Chapter 8) that enable you to log in various formats—for example, log to text files or to an SQL Server table. In instances when the standard logging doesn’t fit your requirements, you can use custom logging, as you’ve done in this Hands-On exercise. You’ve used Execute SQL task to run a parameterized query. If you are wondering about Execute SQL task, it will be discussed many times throughout this book and you will understand all the aspects of its usage. This is one of the main tasks provided in the Control Flow and will be covered in detail in Chapter 5 when we discuss all the preconfigured tasks provided in the Control Flow. User-Defined Variables You can configure user-defined variables for packages, Foreach Loop Containers, For Loop Containers, Sequence Containers, tasks, and event handlers. To configure a user- defined variable, you will provide a name, description, namespace, scope, and value for the variable along with choosing other properties, such as whether the variable raises an event when its value changes or whether the variable is read-only or read/write. Before you begin creating variables, go through the following concepts to understand their creation and usage. Namespace SSIS keeps its variables under namespaces; by default, custom variables are in the User namespace, and system variables are in the System namespace. You can define the 94 Hands-On Microsoft SQL Server 2008 Integration Services custom variables in the User namespace or can create an additional namespace for them, but you cannot modify or add the System namespace or its variables. Scope As mentioned earlier, Integration Services allows variables to be created in the scope of a package or within the scope of a container, task, or event handler in the package. This helps in managing variables, as they are created at the location where they are used. When you create a variable, the scope is selected by the context—i.e., based on your selection of the container or task while you’re creating a variable. You cannot change the scope of a variable after it has been created. The only way to change the scope of a variable is to delete and recreate it in the scope you want. To communicate among objects within a package and to flow the variable values from a parent package to a child package, variables follow these rules: Variables defined at the container scope are available to all the tasks or the child c containers within the container. For example, all the tasks or containers within the Foreach Loop container can use the variables defined within the scope of the Foreach Loop container. Variables defined at the package scope can be used like global variables of DTS 2000, as the package is also a container and is at the top of the container hierarchy. However, for complex packages, it is recommended that you create the variables within the scope where they are needed to help manageability. If a variable has the same name as the variable defined at its parent or grandparent, the c “local” variable will supersede the one descending from the ancestor container. Note that the variables are case-sensitive; therefore, this rule will not affect variables with the same name but with different cased letters. Variables defined in the scope of the called package are never available to the calling c package. SSIS provides a facility to run a package as a child package inside a parent package using the Execute Package task. When a package is run as a child package using the Execute Package task, the variables defined within the Execute Package task scope are available to the called package. at is, the Execute Package task can pass the variables to the child packages. e process of flow of variables from parent package to child package works at run time—that is, the child package variables get values from the parent package when run in the parent package process. If the child package is run on its own, the variable won’t get updated due to the parent not being available. Chapter 3: Nuts and Bolts of the SSIS Workflow 95 Data Type and Value While configuring a variable, you first need to choose the data type the variable is going to be. The value and the data type must be compatible with each other—for example, for an integer data type, you cannot use a string as the value. You can assign a literal or an expression to the value. Evaluating a variable using an expression is a powerful feature that allows you to use dynamic values for variables. Each time the package tasks want to use the variable, they have to evaluate the expression to calculate the variable value. A common use of such a scenario is to load data from the update files received daily, having the date as part of the filename. In this case, you would need to write an expression using the GETDATE() function to evaluate the variable value. Then the expression is evaluated at run time and the variable value is set to the expression result. You can also change variable values using package configurations, which provide the ability for updating values of properties at package loading time as well as package run time. Configurations are useful when deploying packages, as you can update values of variables and connection strings of connection managers. You will study more about configurations in Chapter 13. Hands-On: Creating a Directory with User-Defined Variables Many times, while creating a workflow for SSIS packages, you will need to create folders into which you’ll copy files. In this exercise, you will create a folder using a user-defined variable; this will demonstrate how SSIS can use the values of variables at run time. Method The File System task provided in the Control Flow tab of BIDS does file and folder operations and can create a folder as one of its operations. You will be using this task to create a C:\SSIS\FSTdirectory folder. This exercise is built in two parts: In the first part, you will use static (that is, hard-coded) values in the File System task to create the C:\SSIS\FSTdirectory folder. In the second part, you will use a variable to perform the same task. While the use of variables is extensively covered in Hands-On exercises used for package developments throughout this book, here you will do a simple exercise to add a variable and see how it functions. The File System task is relatively simple to use and configure. This task is covered in detail in Chapter 5, but here the focus is on the use of variables. 96 Hands-On Microsoft SQL Server 2008 Integration Services Exercise (Using Hard-Coded Values) In the first part of this Hands-On exercise, you will create a folder using File System task and directly specifying the folder path and name in the task itself. 1. Start BIDS and choose File | New | Project to create a new project, or press --. The New Project window opens. 2. In the New Project window, choose Integration Services Project from the Templates area. In the Name field type Creating Directory and in the Location field type C:\SSIS\Projects. Click OK to create this new project. 3. After the new project is created, go to the Solution Explorer window and right- click package.dtsx under the SSIS Packages folder. Choose Rename from the context menu. Rename the package as Creating Directory.dtsx and click Yes in the confirmation box. 4. While in the Control Flow designer, open the Toolbox window and drag and drop the File System Task onto the Control Flow designer. You will see a crossed red circle on the task. Hover your mouse on it and it will display a message indicating a validation error. These validation errors are a big help in debugging and creating a syntactically correct package. 5. Double-click the icon of the File System task to open File System Task Editor. With this task, you can create, move, delete, rename, and set attributes on files and directories by selecting one of these operations in the Operation field. Click in the Operation field and select Create Directory from the drop-down list. Selecting this option changes other available fields. This is one of the great features of Integration Services components—to change the available fields dynamically based on the value selected in another field. This way, you can configure only the relevant information in the component rather than getting confused with loads of options. 6. Click in the SourceConnection field to provide the path for the folder and select <New connection…> from the drop-down list. This will open the File Connection Manager Editor. In the previous Hands-On for system variables, you created a connection manager by right-clicking in the Connection Managers area; here you are going to create a connection manager from within a task. Remember that the File Connection Manager allows you to either refer to an existing file or folder or create a new file or a folder. Don’t get confused by the word directory, which is used in File System Task, and folder, which is used in File Connection Manager. They have the same meaning. 7. In the Usage type field, choose Create Folder from the drop-down list. In the Folder field, type C:\SSIS\FSTdirectory. Click OK to return to the File System Task Editor. Note that FSTdirectory has been added in the Connection Managers area and also appears in the SourceConnection field (see Figure 3-7). Click OK to close the File System Task Editor. Chapter 3: Nuts and Bolts of the SSIS Workflow 97 8. Choose Debug | Start Debugging or press 5 to execute the package you’ve developed. The File System task will quickly change to green and will show a “Package Execution Completed” message. The output window also pops up below the Connection Managers area and will show a success message. Press - 5 to stop debugging the package and return to design mode. 9. Run Windows Explorer to see that C:\SSIS\FSTdirectory has been created. Exercise (Using a User-Defined Variable to Pass the Folder Path to File System Task) In this second part of the Hands-On, you will create a user-defined variable and assign a value to it. Then you will assign this variable to SourceConnection field in the File Figure 3-7 Configuring the File System task with hard-coded values . return. Type localhost in the 90 Hands-On Microsoft SQL Server 2008 Integration Services Server Name field. Leave Use Windows Authentication in the Log On To The Server area. In the Connect To. focus is on the use of variables. 96 Hands-On Microsoft SQL Server 2008 Integration Services Exercise (Using Hard-Coded Values) In the first part of this Hands-On exercise, you will create a. 88 Hands-On Microsoft SQL Server 2008 Integration Services Table 3-1 Partial List of System Variables Available in Various Containers System