Quantcast
Channel: SCN : Blog List - SAP HANA Developer Center
Viewing all 676 articles
Browse latest View live

Hana Adapter SDK - Setup

$
0
0

This post is part of an entire series

Hana Smart Data Integration - Overview

 

In order to write new Adapters three things are needed

  1. A Hana instance with the Data Provisioning Server turned on
  2. Eclipse or Hana Studio (preferred)
  3. The Adapter SDK plugin for Eclipse/Hana Studio

 

Enabling the DPServer in the Hana Instance

The first step is done as database administrator by opening the Administration screen of the Hana instance.

adapter sdk setup 1.png

In case the dpserver does not show up there, the configuration tab allows to turn it on (Hana SPS09 or higher).

adapter sdk setup 2.png

 

Eclipse or Hana Studio

If Java SDK 1.7 is installed, the Hana Studio allows to open the Java perspective to develop new Java programs. The Adapter is just yet another Java program based on a couple of interfaces and classes, the Adapter SDK.

The main advantage of using Hana Studio is that SQL commands can be sent to Hana and breakpoints within the adapter being defined in one application. But nothing wrong in using a plain Eclipse installation instead.

 

Adding the Eclipse Plugin for the Adapter SDK

Currently the Adapter SDK is part of the data provisioning agent download, found in service marketplace.

There the various versions for the different Hana releases can be found, the last line is for the latest Hana revision - here Hana SPS10 with the matching SDI 1.00.01 version.

adapter sdk setup 3.png

 

After uncompressing the downloaded SAR file and running the hdbsetup.exe file, a target directory is to be defined and other settings in the next screen.

 

NOTE: After the installation locate the Agent and disable it permanently. We do not want to use the agent with preconfigured adapters, but as part of the development environment.

 

adapter sdk setup 4.png

 

With all components available in this directory, the Adapter SDK can be installed from the "ui" subdirectory, in above installation the c:\apps\dataprovagent\ui folder.

 

In Eclipse or Hana Studio, the menu item Help -> Install New Software, allows to add a local directory as new repository.

adapter sdk setup 5.png

 

In this "ui" directory the Data Provisioning Framework and can be selected for installation.

adapter sdk setup 6.png

 

Once this is installed and the application restarted, a new plugin project can be created, one per adapter. The details are described in the Adapter SDK document found in help.sap.com.

Create a Custom Adapter Using a New Plug-in Project - SAP HANA Data Provisioning Adapter SDK - SAP Library

 

It says there:

Procedure

  1. In SAP HANA studio, open the Plug-in Development perspective (Start of the navigation pathWindowNext navigation stepOpen PerspectiveNext navigation stepOtherNext navigation stepPlug-in DevelopmentEnd of the navigation path).
  2. Right-click in the Package Explorer white space and select Start of the navigation pathNewNext navigation stepPlug-in ProjectEnd of the navigation path.
  3. Enter a project name such as HelloWorldAdapter.
  4. For Target Platform, select an OSGi framework: Equinox and click Next.
  5. Optionally change the Execution Environment depending on your adapter code.
  6. In the Templates dialog, select Data Provisioning Adapter Wizard and click Finish.

 

In case the Data Provisioning Adapter Wizard is not shown in step 6, make sure the step 4 was done - setting the target platform to Equinox.

 

adapter sdk setup 7.png

 

 

The last step is to import the Adapter launch configuration, which is needed to start an adapter within Eclipse.

In the File -> Import dialog under the item Run/Debug a Launch Configuration can be imported from the same "ui" directory.

adapter sdk setup 8.png


Hana Adapter SDK - The first adapter

$
0
0

This post is part of an entire series

Hana Smart Data Integration - Overview

 

 

SAP Hana adapters are Equinox Plugin projects. This means essentially nothing else than writing normal Java code plus an additional Manifest file which contains the dependencies to other plugin projects and libraries.

 

The easiest way to setup all is by using the Eclipse Plugin Development wizard. By calling the File -> New menu item and selecting the type Plug-in Project, the wizard is started.

adapter sdk first adapter 1.png

 

After entering a project name, the important part is to change the plug-in type to OSGi Equinox.

 

adapter sdk first adapter 2.png

 

A Plug-in project consists of certain standard classes like the activator and has a version, again used for dependency handling. Nothing to be changed here.

 

adapter sdk first adapter 3.png

 

Since the OSGi framework was selected before, the Data Provisioning Adapter Wizard is shown and can be selected. Then the created project will have sample implementations of the mandatory classes.

adapter sdk first adapter 4.png

 

At the end the Java/Plugin Perspective will show something like this

adapter sdk first adapter 5.png

A project with the Activator and Factory classes and the real adapter class.

 

Adapters for Hana are available in all kinds of facets

  • Important Adapters are included in the SAP Hana EIM option, e.g. an Adapter for Oracle, SQL Server, DB, etc or SAP ECC and more.
  • Other SAP products require other adapters and they are developed there.
  • We have partners writing new adapters
  • And there is the Open Source approach we just started.

 

The latter is a good starting point for learning how adapters work internally. For example the Adapter HelloWorld used in the Adapter SDK manual is available there.

SAP/hana-native-adapters · GitHub

Hana Adapter SDK - The Open Source Adapters from Github

$
0
0

This post is part of an entire series

Hana Smart Data Integration - Overview

 

In order to import the adapters, the easiest method is using the Github for Windows application.

It is installed from here and provides a fairly simple to use interface for the normal git operations.

Once installed, the github web page with the Hana Native Adapter repository is opened

SAP/hana-native-adapters · GitHub

and by clicking on the Clone in Desktop button a local copy of the github repository is created.

https://help.github.com/assets/images/windows/repository/windows-clone-in-desktop.png

(see here for the official help page of github)

 

This is really just a local copy on the local disk drive with no side effect on the files located in github. The local files can be modified at will.

 

The last step is to go to Eclipse/Hana Studio and import the Existing Project into Workspace using the Files - Import menu item.

adapter sdk github 1.png


As example the HelloWorldAdapter2 can be imported as a first project.

This adapter has no dependencies to other libraries, so it can be started right away. A right click on the Project and selecting the Debug As - Debug Configurations popup menu item does open the next screen.

adapter sdk github 2.png

 

In this screen the AgentConfig Launch Configuration should be visible. If it is not, it can be imported as shown in the previous chapter, by import - Debug - LaunchConfiguration and pointing to the /ui directory of the installed agent.

As every adapter has its own dependencies, the required bundles have to be listed and can be validated if any are missing. But actually, that is quite simple.

  1. Deselect all from the Target Platform tree
  2. Click on the Add Required Bundles
  3. Check if all Bundles in the required version are available by hitting the Validate Bundles button.

 

If the Adapter SDK feature was added as described in the previous chapter and the individual libraries required by each adapter are present, the validation does not return any errors and the Adapter Agent can be started by hitting the Debug button.

 

adapter sdk github 3.png

 

Once started, the console window shows the current information of the running plugins.Using the Stop icon at the top of it, it can be stopped at any time.

adapter sdk github 4.png

 

With these steps the Adapter is running within the Eclipse environment but not known or connected to any Hana instance yet. Basically there are three ways to make the agent and adapter known to Hana.

  1. Execute the create-agent, create adapter etc commands from the SQL console in Hana.
  2. In the Agent install directory run the \configTool\dpagentconfigtool.exe application. It will talk to the Eclipse Agent via TCP and after connecting to Hana it will issue the required Hana commands for registering the agent and the adapters.
  3. In the Console run the command "dplogin hostname port username password". This will create/update the agent using the hostname as agent name and install all adapters started.

Generating mock-data for SAP HANA part 1/3

$
0
0

Today I want to share various variants of generating mock-data on SAP HANA with you. For developing, testing and demonstrating an application it is necessary to fill the connected backend with content. Generating a big amount of data manually can be very time intensive - for saving your time (and nerves ;-) ), this blog post deals with automated data-generation in the SAP HANA environment.

 

Introduction

Before explaining the first procedure to generate data, this section is supposed to give you an overview of how this blog is structured. First of all you have to decide what kind of application you are going to develop. Is it an application that presents data in a static way, that means data isn´t changing often? Or are you going to develop an application in which you realize a lot of changes concerning your stored information?

The following chapter presents various approaches and classifies those applications depending on their "data-changing-frequency".

Following this, the different methods will be explained. First I am going to show you how to create mass data for a HANA database with an external tool (Java application). After that I will go on with two other variants of creating data in a HANA XS environment (for native application development).  A final chapter presents my own opinion and experiences with those differents ways of generating data.

 

Static vs. Dynamic Apps

Now, let´s start classifying your application. The static application deals with data that is not changed that often (for instance, archives, monthly generated data e.g. quarterly results etc.). A dynamic one can be characterized as an application that handles information which are changed often. On the one hand the user wants to see the most up-to-date data and on the other hand it could be changed several times by other (external) events the user cannot influence. As an example for such an dynamic, fast moving application I´d like to mention an Internet-of-Things application. Interacting with "things" or sensors and deriving conclusions out of this delivered data can be very valuable. Especially a demonstration of such an application without having real, phyiscal sensors / "things" is not easy.

Under circumstances, sensors could transmit a lot of different values in a very short time period. To simulate this with mocked-data you have to consider some facts that differ from mocking data for a static application. I am going to come back to this later in the corresponding section.

 

Static: Generating mass data with the Java assistant

 

Requirements:Java required (JDBC connection), DB User with writing rights
Benefits:Mass data generarion, test of HANA processing performance, several generation modes
Downsides:Not automated, not integrated into SAP XS environment

 

Realizing applications on SAP HANA have the advantage of processing mass data in a very efficient way. Handling big amount of data gets done with implementing business logic directly into the database stack. To verify that processing mass data gets done in an appropriate manner you need mocked data for simulation purposes. Generating every single data record in manually way won´t be very efficient. Therefore I´d like to present you a dedicated Java application that is going to help you getting this job done nearly automatically.

 

The following guide leads you through the whole process. A detailed description can be found here.

 

1. After downloading the Java application via this link, a guided procedure leads you through the whole process.

 

2. Connect the Java application to the host your HANA instance is running on.

 

HANA Data Generator 1.jpg

Remark: Please consider that your user you are using for setting up the connection to a HANA instance has sufficient rights for inserting data into the database.

 

3. To find the corresponding database table for inserting randomized mock data, the guide offers you a overview of the whole catalog. This gives you the ability to drill trough the various schemas to the table you are lokking for.

 

HANA Data Generator 2.jpg

 

4. Selecting the table you are willing to insert data, is the next step you have to proceed. Based on the table you have selected the java application prompts you to select the columns in which data has to be written down. Of course all key columns and moreover additional columns which are not allowed beeing "nulled" are selected in advance. The configuration of the mode that is used for creating randomized data completes the setup.

 

HANA Data Generator 4.jpg

The figure above illustrates the different data generation modes (1 to 5). For columns with the data type CHARACTER, two different modes are available. You can choose between the Auto-Generated mode #4 and the Custom mode #5. First mentioned one generates data with the well known "Lorem ipsum..." pattern. The second mode allows you to enter a value which will be written for every data record.

 

Content for date- and numeric-based columns can be created with three various modes. As marked with #1, #2 the Random and the Sequential mode generates values between a lower and an upper value. #3, the Fixed mode, only needs one value which will create nearly similar to #5 afixed, predefined value for the corresponding column for every data record.

The configuration of how many records the tool should create finalizes the generation process. In this last step one more thing has to be considered. Three checkboxes allow additional control for creating a big amount of data.

 

  • Truncate table - if checked, the selected table gets cleared before inserting new data records.
  • Commit each batch - the randomized data gets split up into blocks (size of blocks can be sized as well). This leads to a commit execution after inserting every block (prevents data loss in case of failure).
  • Display progressbar - if checked, this function shows the progress for every batch.

HANA Data Generator 5.jpg

Next week I will show you how to generate data in a HANA XS environment using the oData Explorer, so you don´t have to be dependent on an external toolset.

Have you ever faced the challenge of generating random sample data? If so, let me know how you approached!

 

Stay tuned ;-)



Importing tables from a HANA system into HDFS using Apache Sqoop

$
0
0

This blog is to discuss a way of importing data from an external SAP system with HANA SP10 installed to a system with Red Hat Enterprise Linux using Apache Sqoop. RHEL is the system on which the user is currently logged in using Putty. It can be downloaded from the below link: 
Download PuTTY - a free SSH and telnet client for Windows

Version of Sqoop used is - 1.4.6.2.3.0.0-2557

 

 

 

What happens when a sqoop import is run:

1. Sqoop raises a request for metadata information from the RDBMS system which has the tables in it. JDBC API is used to get the metadata information.

2. The RDBMS system returns the metadata information to the host system. A .java file is created for each table to be imported.

3. The java file is compiled to a .class file and then to a .jar file. The metadata includes the primary key information for each table using which sqoop tries to get the row information from the tables.

4. Data types used in the tables are then converted to respective nearest compatible data types in the HDFS system and imported in the host system.


Internal Commands run during the import:

1. select min(primary_key), max(primary_key) from table_name.

2. select *from table_name where primary_key between min_value and max_val.


SQOOP IMPORT -

1. First step is to copy the ngdbc.jar file from the HANA client directory(C:\Program Files\SAP\hdbclient) to the lib folder inside SQOOP home directory. If you installed Sqoop using Hortonworks, the sqoop directory would be /usr/hdp/hdp-version/sqoop.


2. Change to hdfs user:

su hdfs

where hdfs is the default used of your hadoop system.


3. sqoop import --username <USERNAME> --password **** --connect jdbc:sap://inllxyz12345.abc.global.corp.sap:30015/ --driver com.sap.db.jdbc.Driver --table TABLE_NAME  --hive-import --direct


Above command imports a table named TABLE_NAME from the default schema ADMIN to HDFS.

inllxyz12345.abc.global.corp.sap:3xx15 is given as the system name where xx is the instance number.


--hive-import  -> imports the tables into hive as well as a default directory in the HDFS system.

The table would be imported to hdfs://inllxyz123.apj.global.corp.sap:8020/user/hdfs/SOMETABLENAME

This can be accessed using hadoop fs -ls command


In case the import did not complete, you will have to delete the file created and start import again after resolving issues. Below command is helpful -

hadoop dfs -rm -r hdfs://path/to/file


4. The import was required to be performed from a particular schema, I used SCHEMA_NAME.TABLE_NAME as the --table argument. It gave me the following error -

SQLException in nextKeyValue

Caused by: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [257]: sql syntax error: incorrect syntax near "."

 

Solution: Use the argument currentschema in the host address to make it work as shown below -

sqoop import --username <USERNAME> --password **** --connect jdbc:sap://<host address>/?currentschema=SCHEMA_NAME --driver com.sap.db.jdbc.Driver --table TABLE_NAME --hive-import --direct

 

5. To check if the tables have been moved properly into HIVE, open hive shell:

> hive

> show tables;

displays list of tables

> select * from TABLE_NAME;

 

Other important points/problems and their solutions:

1. In case table name starts with a numeric digit.

Solution - Change the table name to contain only alphabets.


2. If the table name has small case letters.

Solution - The table name should only have capital letters.


3. If no suitable type conversion was possible for a column from the RDBMS to the HDFS.

Solution - Convert the RDBMS column type to a simpler type and try if the import works successfully.


4. DATE type in the RDBMS systems may not be proper.

Solution - Convert the DATE type to a string before importing into the Hadoop system.


5. In case the table does not have a primary key, use split-by argument instead as below:

sqoop import --username <USERNAME> --password **** --connect jdbc:sap://<host address>/?currentschema=SCHEMA_NAME --driver com.sap.db.jdbc.Driver --table TABLE_NAME --hive-import --split-by ID

Here ID is the column name which is used as primary key.


6. If you want to perform parallel map operations so that huge tables are imported in comparatively less time, use "-m n" argument as below:

sqoop import --username <USERNAME> --password **** --connect jdbc:sap://<host address>/?currentschema=SCHEMA_NAME --driver com.sap.db.jdbc.Driver --table TABLE_NAME --hive-import --split-by ID -m 8

8 parallel threads are sun to perform the import.


7. When running the import I also faced below error -

Encountered IOException running import job: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root/.staging":hdfs:hdfs:drwxr-xr-x

Change your user to hdfs (default user for hadoop system).


8. Missing directory error (don't remember exactly but some issue related to missing directory was faced)

Solution:

sudo -u hdfs hadoop fs -mkdir /user/root

sudo -u hdfs hadoop fs -chown root:root /user/root


For any other information related to SQOOP, below user guide can be referred -

https://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html

New Core Data Services Features in SAP HANA 1.0 SPS 10

$
0
0

Core data services (CDS) is an infrastructure for defining and consuming semantically rich data models in SAP HANA. Using a a data definition language (DDL), a query language (QL), and an expression language (EL), CDS is envisioned to encompass write operations, transaction semantics, constraints, and more.

 

A first step toward this ultimate vision for CDS was the introduction of the hdbdd development object in SPS 06. This new development object utilized the Data Definition Language of CDS to define tables and structures. It can therefore be consider an alternative to hdbtable and hdbstructure.

 

In SPS 10 we continue to develop CDS with a focus on expanding the SQL feature coverage and improving complex join operations on views.

 

SQL Functions

 

In SPS 10, CDS is expanded to support almost all of the HANA SQL Functions. This greatly expands the kinds of functionality that you can build into views by formatting, calculating, or otherwise manipulating data with these functions. The following functions are the only ones not yet supported:

  • Fulltext functions
  • Window functions
  • the functions GROUPING, GROUPING_ID, and MAP in the section Miscellaneous function

 

Geo Spatial Types and Functions

 

In SPS 09, CDS first offered support for the usage of the Geo Spatial types in entity definitions. In SPS 10 we expand this support for Geo Spatial in CDS with the addition of GIS functions. This example shows how you can use the function ST_DISTANCE to calculate the distance between two geometry values. Specifically in this example we are taking the address of a business partner which is stored in the database and calculating the distance between it and Building 3 on the SAP Walldorf campus.

 

define view BPAddrExt as select from MD.BusinessPartner {    PARTNERID,    ADDRESSES.STREET   || ', ' || ADDRESSES.CITY   as FULLADDRESS,    round( ADDRESSES.POINT.ST_DISTANCE(                NEW ST_POINT(8.644072, 49.292910), 'meter')/1000, 1) as distFromWDF03   };

Foreign Keys of Managed Associations in Other Associations

In the past using a managed association in a "circular" relationship where the key of entity is used in the association to another entity which in turn uses its key back to the parent would simply have resulted in an activation error. In SPS 10, the compiler now recognizes such relationships. When it sees that the referenced field is actually part of the base entity and thus can be obtained without following the association, it allows activation and doesn't generate any additional columns in the underlying database tables.

 

The following is a common example of just such a Header/Item relationships:

entity Header {  key id : Integer;  toItems : Association[*] to Item on toItems.head.id = id;
};
entity Item {  key id : Integer;
 head : Association[1] to Header { id };
};

Unlike a normal managed association, no additional column is generated for the association in the underlying database table. So this case it acts very much like an unmanaged association.

header.png

 

Filter Conditions

Another new features in SPS 10 is the addition of filter conditions. When following an association, it is now possible to apply a filter condition which is mixed into the ON-condition of the resulting JOIN. This adds more power and flexibility to the views you can build via CDS while also following the idea of CDS to make the definition more human readable and maintainable than the corresponding pure SQL functionality.

 

In this first example we apply a simple, single filter on LIFECYCLESTATUS to the Business Parnter -> Sales Order join.

 

 

view BPOrdersView as select from BusinessPartner {  PARTNERID,  orders[LIFECYCLESTATUS='N'].SALESORDERID as orderId
};

The resulting generated view is:

view1.png

Associations with filters are never combined.  Therefore in order to tell the compiler that there actually is only one association, you have to use the new prefix notation. In this example we want the LIFECYCLESTATUS filter apply to both the SALESORDERID and GROSSAMOUNT retrieval via association.

 

view BPOrders2View as select from BusinessPartner {  PARTNERID,  orders[LIFECYCLESTATUS='N'].{ SALESORDERID as orderId,                                GROSSAMOUNT  as grossAmt }
};

The resulting generated view is:

view2.png

But we also see that by using the prefix notation, that such filters can be nested. This example expands on the earlier one. It still filters business partners who only have orders with LIFECYCLESTATUS = N, but now also only selects those who have ITEMS with a NETAMOUNT greater than 200.

 

view BPOrders3View as select from BusinessPartner {  PARTNERID,  orders[LIFECYCLESTATUS='N'].{ SALESORDERID as orderId,                                GROSSAMOUNT  as grossAmt,                                ITEMS[NETAMOUNT>200].{ PRODUCT.PRODUCTID,                                                       NETAMOUNT }                              }
};

The resulting generated view is:

view3.png

 

Series

The final new feature in CDS to discuss today is Series. Series data allows the measuring of data over a time where time is commonly equidistant; it allows you to detect and forecast trends in the data. You can read more about the general functionality of Series data in SAP HANA here: http://help.sap.com/hana/SAP_HANA_Series_Data_Developer_Guide_en.pdf

 

The major addition from the CDS side is that you can define Series data within CDS entities.  Here is a small example of the use of the series keyword:

entityMySeriesEntity{  key setId : Integer;  key t : UTCTimestamp;
 value : Decimal(10,4);  series (
series key (setId)
period for series (t)          
equidistant increment by interval 0.1 second
)

};

Generating mock-data for SAP HANA part 1/3

$
0
0

Today I want to share various variants of generating mock-data on SAP HANA with you. For developing, testing and demonstrating an application it is necessary to fill the connected backend with content. Generating a big amount of data manually can be very time intensive - for saving your time (and nerves ;-) ), this blog post deals with automated data-generation in the SAP HANA environment.

 

Introduction

Before explaining the first procedure to generate data, this section is supposed to give you an overview of how this blog is structured. First of all you have to decide what kind of application you are going to develop. Is it an application that presents data in a static way, that means data isn´t changing often? Or are you going to develop an application in which you realize a lot of changes concerning your stored information?

The following chapter presents various approaches and classifies those applications depending on their "data-changing-frequency".

Following this, the different methods will be explained. First I am going to show you how to create mass data for a HANA database with an external tool (Java application). After that I will go on with two other variants of creating data in a HANA XS environment (for native application development).  A final chapter presents my own opinion and experiences with those differents ways of generating data.

 

Static vs. Dynamic Apps

Now, let´s start classifying your application. The static application deals with data that is not changed that often (for instance, archives, monthly generated data e.g. quarterly results etc.). A dynamic one can be characterized as an application that handles information which are changed often. On the one hand the user wants to see the most up-to-date data and on the other hand it could be changed several times by other (external) events the user cannot influence. As an example for such an dynamic, fast moving application I´d like to mention an Internet-of-Things application. Interacting with "things" or sensors and deriving conclusions out of this delivered data can be very valuable. Especially a demonstration of such an application without having real, phyiscal sensors / "things" is not easy.

Under circumstances, sensors could transmit a lot of different values in a very short time period. To simulate this with mocked-data you have to consider some facts that differ from mocking data for a static application. I am going to come back to this later in the corresponding section.

 

Static: Generating mass data with the Java assistant

 

Requirements:Java required (JDBC connection), DB User with writing rights
Benefits:Mass data generarion, test of HANA processing performance, several generation modes
Downsides:Not automated, not integrated into SAP XS environment

 

Realizing applications on SAP HANA have the advantage of processing mass data in a very efficient way. Handling big amount of data gets done with implementing business logic directly into the database stack. To verify that processing mass data gets done in an appropriate manner you need mocked data for simulation purposes. Generating every single data record in manually way won´t be very efficient. Therefore I´d like to present you a dedicated Java application that is going to help you getting this job done nearly automatically.

 

The following guide leads you through the whole process. The pictures and also some terms presented below are taken from a document created by Erik Lemen. For comparison reasons with other approchaes of data generation I´d like to summarize parts of this guide. More detailed information about this process can be found in Erik´s guide.

 

1. After downloading the Java application via this link, a guided procedure leads you through the whole process.

 

2. Connect the Java application to the host your HANA instance is running on.

 

HANA Data Generator 1.jpg

Remark: Please consider that your user you are using for setting up the connection to a HANA instance has sufficient rights for inserting data into the database.

 

3. To find the corresponding database table for inserting randomized mock data, the guide offers you a overview of the whole catalog. This gives you the ability to drill trough the various schemas to the table you are lokking for.

 

HANA Data Generator 2.jpg

 

4. Selecting the table you are willing to insert data, is the next step you have to proceed. Based on the table you have selected the java application prompts you to select the columns in which data has to be written down. Of course all key columns and moreover additional columns which are not allowed beeing "nulled" are selected in advance. The configuration of the mode that is used for creating randomized data completes the setup.

 

HANA Data Generator 4.jpg

The figure above illustrates the different data generation modes (1 to 5). For columns with the data type CHARACTER, two different modes are available. You can choose between the Auto-Generated mode #4 and the Custom mode #5. First mentioned one generates data with the well known "Lorem ipsum..." pattern. The second mode allows you to enter a value which will be written for every data record.

 

Content for date- and numeric-based columns can be created with three various modes. As marked with #1, #2 the Random and the Sequential mode generates values between a lower and an upper value. #3, the Fixed mode, only needs one value which will create nearly similar to #5 afixed, predefined value for the corresponding column for every data record.

The configuration of how many records the tool should create finalizes the generation process. In this last step one more thing has to be considered. Three checkboxes allow additional control for creating a big amount of data.

 

  • Truncate table - if checked, the selected table gets cleared before inserting new data records.
  • Commit each batch - the randomized data gets split up into blocks (size of blocks can be sized as well). This leads to a commit execution after inserting every block (prevents data loss in case of failure).
  • Display progressbar - if checked, this function shows the progress for every batch.

HANA Data Generator 5.jpg

Next week I will show you how to generate data in a HANA XS environment using the oData Explorer, so you don´t have to be dependent on an external toolset.

Have you ever faced the challenge of generating random sample data? If so, let me know how you approached!

 

Stay tuned ;-)



Hana Adapter SDK - Interaction via SQL

$
0
0

This post is part of an entire series

Hana Smart Data Integration - Overview

 

 

 

There are multiple UIs like the Web based FlowGraph editor and ReplicationEditor or Hana Studio available to setup and use the Adapters. But under the cover they are all executing SQL commands. Hence it might be a good idea to look at the adapters from that angle as well.

 

Recap - SQLs for creating the agent and adapter

In the previous chapters the agent and the adapter was registered in Hana. These applications use SQL regular commands, in particular

 

CREATE AGENT "mylaptop" PROTOCOL 'TCP' host '10.16.94.83' PORT 5050; (see CREATE AGENT - SAP Library)

CREATE ADAPTER "HelloWorldAdapter" AT LOCATION AGENT "mylaptop"; (see CREATE ADAPTER - SAP Library)

 

If the IP address changed or one adapter is hosted at another location, there are the matching ALTER commands available as well.

 

To check which agents and adapters are available, the Hana data dictionary views AGENTS, ADAPTERS, ADAPTER_LOCATIONS can be queried.

 

Up to now, the adapter - let's call it the middleware - was installed and Hana made aware of it.

 

 

Creating the source connection

 

In order to actually use an adapter, a remote-source has to be created. This provides all the login information the adapter needs to connect to a particular system using the adapter as a driver.

 

 

CREATE REMOTE SOURCE "myHelloWorldConnection" ADAPTER "HelloWorldAdapter"

AT LOCATION AGENT "mylaptop" CONFIGURATION

  '<?xml version="1.0" encoding="UTF-8"?>

  <ConnectionProperties name="conn">

    <PropertyEntry name="name">Myself</PropertyEntry>

  </ConnectionProperties>'

WITH CREDENTIAL TYPE 'PASSWORD' USING

  '<CredentialEntry name="credentials">

    <user>me</user>

    <password>unknown</password>

  </CredentialEntry>';

 

The information what parameters it needs and what their names are is stored in the ADAPTERS data dictionary table, like for above adapter it reads:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<RemoteSourceDescription>

    <ConnectionProperties displayName="Connection Info" isRepeatable="false" name="conn">

        <PropertyEntry displayName="Hello whom?" id="1" isRepeatable="false"

          isRequired="false" name="name" type="STRING"/>

    </ConnectionProperties>

    <CredentialProperties displayName="Credentials" name="credentials">

        <CredentialMode displayName="Credentials Mode" id="2" isRepeatable="false"

          isRequired="false" name="credentials_mode" type="ENUM">

            <EnumValue displayName="Technical User" name="technicaluser"/>

            <EnumValue displayName="None" name="none"/>

            <EnumValue displayName="Secondary Credentials" name="secondarycredentials"/>

        </CredentialMode>

        <CredentialEntry displayName="Credentials" id="3" name="credential">

            <user displayName="Username" id="4" isRequired="true" type="STRING"/>

            <password displayName="Password" id="5" isRequired="true" type="STRING"/>

        </CredentialEntry>

        <Activator id="2" name="credentials_mode">

            <Value>technicaluser</Value>

        </Activator>

    </CredentialProperties>

</RemoteSourceDescription>

 

 

But frankly, using Hana Studio -> Provisioning from the Catalog view is much easier.

adapter sdk sql 1.png

 

 

Adding virtual tables

 

A virtual table can be seen as a database View inside Hana on top of a remote table to blend this table into the Hana dictionary seamlessly. Hence the syntax is quite simple:

 

create virtual table v_hello at "myHelloWorldConnection"."<NULL>"."<NULL>"."HELLO";

 

The more interesting question is how to know the values for database, owner and table name. Since these are remote tables, the only place to get the information from is the adapter itself. Using the procedure GET_REMOTE_SOURCE_OBJECT_TREE the adapter can be queried.

 

call SYS.GET_REMOTE_SOURCE_OBJECT_TREE ('myHelloWorldConnection', '', ?, ?);

 

adapter sdk sql 2.png

 

But here as well, using the Catalog Editor might be simpler.

adapter sdk sql 3.png

 

But the create virtual table statement allows to pass parameters to the adapter as well. This is for example used with the file adapter. A virtual table for a file needs lot of knowledge, what the delimiter is, what the first column should be named and its datatype etc. Hence a virtual table for the file adapter can be created with a SQL like this:

create virtual table v_fixed at "D050767_FileAdapter"."<NULL>"."<NULL>"."fixed"

REMOTE PROPERTY 'dataprovisioning_parameters'='<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<Parameters>

<Parameter name="FORMAT">FIXED</Parameter>

<Parameter name="FORCE_FILENAME_PATTERN">fixed%.txt</Parameter>

<Parameter name="ROW_DELIMITER">\r\n</Parameter>

<Parameter name="SKIP_HEADER_LINES">1</Parameter>

<Parameter name="COLUMNSSTARTENDPOSITION">0-1;2-7;8-15</Parameter>

<Parameter name="ROWLENGTH">16</Parameter>

<Parameter name="COLUMN">COL1;NVARCHAR(2)</Parameter>

<Parameter name="COLUMN">COL2;NVARCHAR(6)</Parameter>

<Parameter name="COLUMN">COL3;NVARCHAR(8)</Parameter>

<Parameter name="CODEPAGE">UTF-8</Parameter>

</Parameters>';

 

The select command

A select command can be used to get data from a virtual table. What part of the user SQL is sent to the Adapter is controlled by the AdapterCapabilities, the remaining logic is executed in Hana. Well, that's the Smart Data Access federation logic.The interesting part for us as adapter developers is that we can again send additional information to the adapter as part of the select statement.

File adapter is a good example for that again:

SELECT * FROM V_FILECONTENTTEXT WHERE NAME LIKE 'fixed1.txt'

WITH DATAPROVISIONING PARAMETERS

( '<Table name="V_FILECONTENTTEXT"><Parameter name="CODEPAGE">ISO-8859-1</Parameter></Table>' );

 

The main difference to above is that the select statement can contain multiple tables potentially and then the SQL parser needs to know which parameter belongs to what table. Hence the SQL contains the virtual table name as a parent XML node.

 

The adapter parameters

The adapter itself supports setting parameters as well. These are not part of any SQL command, it is part of the startup code of the adapter process. Therefore it is set by using the configTool/dpagentconfigtool.exe shipped as part of the data provisioning agent install.

The HelloWorldAdapter does not require any parameters to be set but the FileAdapter lets you enter the root directory where all the files are located, in order to make sure nobody writes e.g. into the c:\Windows directory.

adapter sdk sql 4.png


Generating mock-data for SAP HANA part 2/3

$
0
0

As promised last week,with part two of my blog series I’m going to show you how to generate data within the HANA XS environment.


Static: Data generation with the OData Explorer in HANA XS


Requirements:DB user with writing rights, XS application development
Benefits:Integrated into HANA XS environment and into Web-based Development Workbench, Fill the backend with data for demonstrating purposes
Downsides:Not automated, Execute the procedure once generates only <= 10 data records, For generating more data records a repeated execution is necessary

 

Developing native HANA applications has various advantages in terms of integration of data storage and data processing. For instance oData is used as a common approach to expose data to front-end applications using UI5. With the help of configuration files (.xsodata), stored data in corresponding tables and views can be accessed. Obviously it is necessary to have some data in the database ;-). Creating content in the XS framework could be done with the help of the xsodata file itself and the OData Explorer. Using this tool, which is based on the SAP River Application Explorer, gives you the possibility to do this. Some of you will probably remember, it was released within SPS07.

 

Let´s get started and generate some mock-data with the OData Explorer!

 

1. Choose the appropriate .xsodata file, which contains the table or view you´d like to generate data for. The tool uses the metadata as template and proposes the columns for data generation.

 

oData_Explorer1.jpg

 

2. Right-click on this file. A context-menu opens. Choose the menu point "Open OData Explorer". Afterwards the explorer opens and displays all repository objects that are configured for being exposed with this file (in this case only one view).

 

oData_Explorer2.jpg

3. After selecting the desired repository object (in this case the selection is quite limited ;-) ) you have to press the "Generate Data" button as indicated above. A dialog opens and prompts you to configure the data generation process.

 

oData_Explorer3.jpg

You´ve got the choice between three different data generation modes. #1 shows the Fixed mode. You just have to enter a value that sticks to the defined data type and afterwards this value will be inserted for the selected amount of data records - maximal 10! #2 presents the No value mode. Selecting this one leads to no data generation for this column. The last mode (#3) creates data in a random manner.

 

Due to the fact I haven´t found any information about how data is generated by using the random mode and which algorithm(s) are involved I´d like to show you the result of this automated data generation process.

 

Generate_Data.png

As this figure shows, data for columns with random mode configuration gets inserted by using a pattern. Columns defined as data type "CHAR" use the name of itself as prefix followed by numbers generated randomly. Columns whose definition differs from "CHAR" generate data in a random manner without using any prefixes. I´m aware giving you more detailed information about this would be more satisfying - so if anybody knows more about this: Those information are much appreciated. :-)

 

This way of generating data is more appropriate for inserting data sporadically into the database than generating mass data. Giving the customer/user an idea of how the application works could be one of the use cases in which you are going to use this tool. In my opinion it is not suited for getting an idea how the application handles big data as it does not create enough data records at once.

 

My last blog post connected to this series is going to deal with generating data records for dynamic applications. Hints or best practices how to achieve this goal are very welcomed! Next week, I´m going to present my approach to mass data generation for dynamic applications. Stay tuned!

Integrate maps on SAP UI5 developments

$
0
0

Hello Developers,


This post will show how to integrate maps on SAP UI5 developments. It gets interesting when a GoogleMaps service comes into play, it's pretty simple though.


I've found myself with an application requirement that consisted of displaying addresses in maps. Thanks to SAP UI5 developers we now have a library for maps rendering: sap.ui.vbm.GeoMap. However, this library expects coordinates but my customer database has human-readable addresses like Carrera 7 # 28-66 Bogotá. Here is what the binding looks like into sap.ui.vbm.GeoMap:


vos: [        new sap.ui.vbm.Spots({             //click: onClickSpots,             //contextMenu: onContextMenuSpot,             // bind to an array in the model             items: {             path : "/data",                  template: new sap.ui.vbm.Spot({                  position: '{Coordinates}',                  tooltip: '{CardName}',                  image: 'pin_blue.png',                  })              }        }),
]

 

Take a look at the position element: it requires the coordinates which I didn't have.

 

So, I've been to the Google Developers web site and found this: https://developers.google.com/maps/documentation/geocoding/intro, a geocoding API that converts addresses to coordinates. It's free to use with some restrictions and very easy to implement, but I'leave the explanations to the Google guys.

 

The following code implements the call to the Google service which expects the address I'd like to convert and a key (provided by Google) to be passed as parameters:

 

var oGoogleModel = new sap.ui.model.json.JSONModel('https://maps.googleapis.com/maps/api/geocode/json?address='+address+'&key='+key);
oGoogleModel.attachRequestCompleted(oModel,function(oEventG){  var oModelG = oEventG.getSource();  var data = oModelG.getData().results[0].geometry.location;  // After this coordinates are stored in data.lng and data.lat
});

 

As shown in the code above, the Google API returns a json array with lots of information including the required coordinates.

 

Hope this was helpful.

 

Alejandro Fonseca

Twitter: @MarioAFC

Hana Adapter SDK - The Adapter code

$
0
0

This post is part of an entire series

Hana Smart Data Integration - Overview

 

 

 

 

When looking at the adapter code, there are several dimensions to consider:

  • The hierarchy of classes
  • The Adapter Capabilities
  • The call hierarchy
  • The relationship between user actions and the API being called

 

Class Hierarchy

 

The hierarchy of the classes is

  1. public abstract class com.sap.hana.dp.adapter.sdk.Adapter
  2. public abstract class com.sap.hana.dp.adapter.sdk.AdapterCDC extends com.sap.hana.dp.adapter.sdk.Adapter
  3. public abstract class com.sap.hana.dp.adapter.sdk.adapterbase.BaseAdapterClass extends com.sap.hana.dp.adapter.sdk.AdapterCDC

 

The Adapter class is the bare minimum, to be used when writing an adapter for SDA only, no realtime. If realtime should be supported as well, then the AdapterCDC class has to be the foundation.

Both of these classes are very low level. In order to build an adapter based on those, a lot of common code has to be written.

That is the reason why the BaseAdapterClass was built in top of it, which has default implementations for most tasks, which leaves only the bare minimum to be implemented. Use this class as the foundation whenever the adapter is no database, so it does not support joins, only simple where clauses, no group by.

Hence for database adapters the common starting point will be Adapter oder AdapterCDC, for all other adapters the BaseAdapterClass. Even if the adapter does not support Realtime, the BaseAdapterClass can be used still - see Capabilities.

 

Capabilities

 

What the adapter can or cannot do is not controlled by the classes implemented but by returning certain things in the getCapability() method. So even if the AdapterCDC or BaseAdapterClass is being used, the adapter has to tell Hana that is supports realtime by returning a list of capabilities containing the element AdapterCapability.CAP_TRANSACTIONAL_CDC in above method.

The different types of Capabilities are a topic of its own but in general terms, they control what kind of SQLs are supported. If the adapter does not support realtime, Hana will raise an error whenever the command create-remote-subscription is executed using this adapter or a virtual table of it.

see Capabilities - SAP HANA Data Provisioning Adapter SDK - SAP Library

 

Call Hierarchy

 

Every Hana sessions opens a new Hana adapter instance. So as seen by Java, an Adapter object is created and its methods being called depending on what SQL statements the user executes.

This can be tested easily by setting breakpoints at every adapter method and starting the adapter in Eclipse in debug mode - this has been discussed in the previous chapters already.

For a better understanding the table below shows the action done by the user and its effect on the adapter using the HelloWorld2Adapter as example, the one that is extending the BaseAdapterClass.

 

Prep Calls

 

The first stage is when registering an adapter in Hana. Then the adapter's core metadata is queried and stored in the Hana data dictionary table. Because of that any change in the adapter properties and capabilities require the adapter to be refreshed.

 

SQLs:

CREATE ADAPTER "HelloWorldAdapter2" AT LOCATION AGENT "mylaptop";

ALTER ADAPTER "HelloWorldAdapter2" REFRESH AT LOCATION AGENT "mylaptop";


Messages exchanged between DPServer and Adapter:

  • GET_CAPABILITIES
  • MDS_REMOTE_SOURCE_DESC


The Remote Source Description is an XML consisting of two kinds of property trees, the regular parameters that are public and credential infomration which has to be stored in the secure store by Hana.


The method addRemoteSourceDescriptors(PropertyGroup root) is called and the adapter adds a tree of nodes to it, which will be turned in parts of the remote source XML internally.

adapter sdk calls 1.png

 

Similar the method addRemoteSourceCredentialDescriptors(CredentialProperties credential), where the adapter adds all the nodes which should end up in the secure store of Hana. So essentially both methods do the same thing, they together form the XML being returned, but since the credential tree has to follow more strict rules, it is a separate call with its own helper methods.

 

adapter sdk calls 2.png

 

 

Open and Close

 

With above information the UI has the knowledge what parameters the user has to supply when creating a concrete remote source connection using the adapter.

adapter sdk calls 3.png

These entered values are sent to the Adapter's open() method in form of a RemoteSourceDescriptor and can be used to e.g. connect to the actual source system using the supplied information.

Whenever a session is no longer needed, a timeout happens or any other error in the connection between Hana and the Adapter, the corresponding close() method is called,

 

adapter sdk calls 4.png

 

Browsing

 

When browsing for tables, e.g. by expanding the created remote source, a session is opened and the addNodes() method being called.

 

adapter sdk calls 6.png

 

The task of the adapter is to add child nodes which should be rendered in the UI then. These returned nodes can either be tables, like in this example a TableBrowseNode representing a certain table is added, or it can be a group node. Such group node can then be further expanded in the UI. In order to know if the root node or such child node was expanded, the method getParentNodeIDString() returns either NULL (the root node got expanded) or the unique name of a group node. There are some more methods to understand the depth of the current tree and the such.

 

adapter sdk calls 5.png

 

The Table Metadata

 

When a create virtual table is issued, either manually or via the UI, the adapter's importMetadata() method is invoked. In the BaseAdapterClass an empty table object is created already and the task of the adapter developer is to provide the table name, the columns with datatypes and the such.

Since there could be multiple tables, the table to import is provided in the first parameter. When the browsing is a flat list of table only, this ArrayList is just one element large and contains the table name itself, the same name as being used in the browsing method above.

But if the browsing is multiple levels, this ArrayList contains the path to the unique object, e.g. rootdirectory, subdirectory1, subdirectory2, filename.

 

adapter sdk calls 7.png

 

In the BaseAdapterClass all handling of one table or a set of tables of similar kinds is handled in a second class, the TableLoader, for better reading. The example code for the HelloWorldAdapter2 is:

public static void importMetadata(TableMetadata table) {

  table.setName(HelloWorldAdapter2.HELLO);

  table.setPhysicalName(HelloWorldAdapter2.HELLO);

  table.setDescription("Hello World table");

 

 

 

  addColumnInteger(table, "ROWNUMBER", "The row number");

  addColumnNVarchar(table, "TEXT", 80, "The Hello text");

}

 

The table gets a name and columns using these or similar helper methods.

For a database adapter the corresponding code might be different. One TableLoader is covering all tables of the database and the adapter does read the source database data dictionary in order to return the requested table structure. The adapters main task would be to assign the correct Hana datatypes for each source column datatype.

 

Execution of a query

 

When the BaseAdapterClass receives the request to execute a query, this higher level class has implemented the code to break apart the sent SQL text into its components using a SQL Parser the Adapter SDK provides. As the BaseAdapterClass does not support any joins, this class can simplify a lot of things. For example it creates a new TableLoader class for this particular table and provides all information like the table name, the where clause being supplied etc.

 

The TableLoader has methods to override so it can for example execute code against the source system.

 

The task of the TableLoader implementation is to return one object in the getNextRowData() which contains all the information of one row. This method is called until it tells that there is no more data available.

This object is then passed into the setColumnValue() method along with the information. The reason for this two staged approach are projections.

Imagine above table, it has the two columns ROWNUMBER and TEXT. The SQL being executed selects just the column ROWNUMBER. Or it might be a statement like

select TEXT, ROWNUMBER, ... from ...

The adapter needs to assign to these three columns the proper source value, do all the conversions and the such.

Therefore the setColumnValue() method is called once for every column of the select(!) statement and it provides the information which column should be set and what its source column is.

 

@Override

protected void setColumnValue(

    int tablecolumnindex,

    int returncolumnindex,

    AdapterRow row,

    Object o) throws AdapterException {

  switch (tablecolumnindex) {

    case 0:

      row.setColumnValue(returncolumnindex, (Integer) o);

      break;

    case 1:

      row.setColumnValue(returncolumnindex, ... + "Hello" + ...);

      break;

  }

}

 

For the select TEXT, ROWNUMBER from ... this method would be called three times with the same Object o as input and the the values of the pair would be

returncolumnindextablecolumnindexSource Table Column
01TEXT
10ROWNUMBER

 

Return column index is a simple counter, 0..first column of the select, 1..second column etc and the tablecolumnindex is the index of the column as being created in the virtual table. The virtual table has at index 0 the column ROWNUMBER, then the TEXT column was added hence got the index 1.

 

adapter sdk calls 8.png

 

 

adapter sdk calls 9.png

 

 

It has to be noted that the SQL statement sent to the adapter is NOT(!) the SQL statement being entered in the query. The federation logic of SDA is inbetween, which tries to optimize the SQL sent and also makes sure the adapter capabilities are honored.

 

For example this select statement here returns the correct result.

adapter sdk calls 10.png

 

Which might be surprising at first, since nobody implemented the substring function in the adapter. When looking at the explain plan for this statement, it becomes obvious what happens.

adapter sdk calls 11.png

The SDA layer rewrote the statement to something like

 

select text, rownumber, text, substring(text, 1, 1) from

  (select ROWNUMBER, TEXT from v_hello);

 

The inner part being executed by the adapter, the remaining logic inside Hana. The other point noteworthy is the aliasing of the SQL being sent to the adapter. The from clause reads: from "HELLO" "V_HELLO"

So the SQL provides the name of the virtual table - which the adapter does not care about, but also the name of the remote table the adapter is using.

Generating mock-data for SAP HANA part 3/3

$
0
0

The blog post published last week deals with generating of mock-data within the XS engine for static applications. Finally with this last post, our journey through different ways of generating data on HANA ends. Last but not least I´d like to show you how to approach data generation for dynamic applications (e.g. an IoT application).

 

Data generation with the XS Scheduler and an own XSJS algorithm in a HANA XS environment

 

Requirements:XSJS coding skills, Rights to access the involved schemas and the corresponding database tables
Benefits:

Interact with the HANA database in a very easy manner, Flexibility concerning data generation (it´s you who creates the generation algorithm),

Combines advantages of part 1 and part 2, Avoids using external tools

DownsidesYou have to code everything on your own --> can result in more complexity, Consumption of HANA ressources for data generation may slow down your XS instance

 

Data generation for a dynamic application requires the creation of data in a flexible way. However it is necessary to create data records over and over again.

Therefore the HANA XS environment provides a framework component called XS Scheduler.


It allows you to schedule the execution of XSJS programms once or several times (e.g. every minute). This gives you the opportunity writing your own data generation algorithm and following this, simulating a change of sensor data. It leads to a more exciting experience for your audience watching the application in action.

With following figure I´d like to visualize this concept and show you how I have realized data generation for an IoT application.


XSScheduler.png

 

Own experiences and recommendations from my side

 

Using one of the first two variants for generating data (for a static application) is completely sufficient. Please keep in mind whether you want to verify the performance of your application or not ( --> For veryfying purposes I´d prefer to use the first Java application).

Creating and simulating an Internet-of-Things scenario requires another proceeding. For me the way of using the XS Scheduler was the only possible and most easiest one. Another thinkable alternative would be using Java, to code your own external application, which avoids the consumption of HANA XS ressources.The usage of the JDBC interface gives you the possibility to connect it to the HANA database. However you need coding skills in this language. Be aware of hosting this application on a different framework/host than your HANA XS might be necessary (Yes, there is a possibility to deploy Java applications on SAP HANA. For a better controlling of the data generation process I´d suggest hosting it on a client, spearated from HANA). Under circumstances this might be more difficult than using the HANA XS Scheduler and the solution I´ve used.

 

All in all, HANA offers you several ways you can use for achieving your goal in generating mock-data. It´s up to you to try them out and find the solution that fits to your scenario most. I hope I could give you a short but sufficient overview about different approaches . If you have any questions concerning this blog post series, please don´t hesitate to contact me ... ;-)

SAP Chat System through all interface...

$
0
0

Hi,

 

I have posted an IDEA on SAP Idea Incubator, about Chat System in SAP through all Interface like SAP GUI, UI5..


SAP Chat System : View Idea


It's same like other chat system but one thing i have to say..we are working in business world and sharing business information through social chat application is not at all suggestible. That's the idea to share the business information like document no, business mail id's, any other business related information though inside business application.


I have made an application in SAP Netweaver platform, where two or multiple logged in user can share the information through SAPGUI chat application.


Please find below link, where i have given all initial step about the application.


The same application i have started to build in SAPUI5 platform, because now a days most of the users are using FIORI application through there hands on devices.



.Let's chat...



info.jpg

 

Additional information would be more help to add some more feature in application.

 

Thanks & Regards.

Praveer.

Accessing HANA Cloud developer edition from Amazon web services

$
0
0

Introduction


Intention of this blog is to help people who want to get hands on to HANA Development using Amazon web services (AWS). I have explained the step by step process as on date of  publishing this blog to get connected to an HANA instance on AWS. Those who are referring to the blog, please keep in mind that cloud service charges I mentioned below will be fluctuating from time to time, so make sure the charges falls in your budget .



Setting up an Amazon (AWS) Account


  • You will require an Amazon account for using Amazon web services(AWS), for that go to [  https://aws.amazon.com/ ]. If you already have an amazon account use that account else create a new account, you will have to furnish your credit card / Debit card details here. Your monthly invoice amount will be detected from this account. Make sure you enter valid details here.
  • As part of validating your account details AWS will charge you a small amount ( 2 INR for me)
  • There will be IVR verification of your mobile number.Once verified you will be prompted to select a support plan, as our usage is purely training you can go for Basic support plan which is free.
  • Proceed to AWS Management Console
  • Click on your Account name and from the drop-down select Security credentials

11.jpg

  • In this screen expand the access keys and “Create new access key”. This information is important, because anyone who knows this information can access your training system which will incur you charges. Once you press this button, it will generate access key and will ask you to save it as text file, save this information securely as this information will be required by you whenever you wanted different services from AWS.
  • You will get an Access key and secret key with the above action which will be used for configuring your solution

 

Set up Hana Instance in Cloud appliance Library


  • Use your SAP Community Network credentials [ if you dont have a SCN user go to https://www.scn.sap.com/ and signup your self ] to logon to SAP Cloud Appliance Library[ https://cal.sap.com ]
  • Go to Accounts tab


  • On the right corner of the above you can see “Create account” Press this button

 



  • Select the cloud provider as Amazon Webservices in the next screen and provide the Access key and Secret key what you got from AWS while registration here for setting up the account ( You can leave the cost assignment field ) and press finish
  • Note down your SCN user id for future references ( P*****)
  • Once you finish you can see your username configure in the accounts console on SAP cloud appliance library

 

 

  • Now go to the solution tab and search for “SAP HANA developer edition”. Activate / Trynow  the latest SPS available
  • It will show as create Instance, where you have to press the create instance button
  • Once you press create instance it will show you the following screen with pricing information, the highlighted amount is the charges applicable. Amazon will charges you additional for keeping you instance data on cloud as per their pricing [ As per my experience it was 0.48 USD + 1 USD  which  will cost you around USD 1.5  approximately 100 INR per hour. But this is dependent on how many objects you develop, I recommend you to store as less objects as possible to keep the charges on your training down

 

  • In the next screen you will be able to see the access, in which you can see the ports HANA use for connecting to cloud server. So please make sure the ports are open otherwise you will find it difficult in connecting to HANA Instance

 

 

  • Set your solution password, these will be used for your HANA studio  login

 

 

  • In the Schedule tab select Manually Suspend & Activate
  • Once you finish, it will ask to save the private instance key. Save it securely where you can access it later also
  • It will take 20 mins for your HANA system to get set up right and running
  • Once the instance is ready it will show a green icon, the click on your Instance name

 

 

  • Now you can see the server IP details as below

 

  • Install HANA Studio on your PC [ Refer the link for HANA studio installation SAP Development Tools for Eclipse ]
  • Open your HANA Studio and go to system tab, right click and click add system

 

 

  • Press Next & Give the user name as SYSTEM & Password as the solution password you set earlier in cloud appliance library and press finish


 

  • Once your instance is ready to get started, you can see the added server as green or yellow ( Some time due to error in initiating web services the instance show yellow, but there is nothing to worry, it won’t stop your from doing the modelling / creating views )



NB : You should always take care that once you are done with your development practicing, you must go to your CAL account and "SUSPEND" the instance else if you simply log off from hana studio, AWS will charge you for the time till you suspend the instance. There will be also a "TERMINATE" button against your instance, if you wish NOT TO store your objects you can use this which will delete the HANA instance, next time you have to set up a new one.


Also each time when you activate your instance the External IP will change. So you will have to go to your studio and change the IP of the added system. If you like you can opt for static IP while setting up your instance but there will be again charges involved for the static IP.


Let me know your feedback on how helpful was this blog -

Happy learning

Deric Pavoo

Migrating a HCP MaxDB database to HANA

$
0
0

Overview: This blog describes migrating a simple MaxDB database from HCP to a dedicated HCP HANA instance. Also has some details around a open source tool called XSImport - a free, open source native XS app for importing CSV files into HANA.


HCPMaxDBtoHANA.png

In the Enterprise Mobility team at SAP we are currently working on an exciting project migrating one of our existing applications to a HANA backend, while going through a major code rewrite. During this process, I had the task of migrating a HCP based MaxDB over to a dedicated HCP HANA DB. The dataset I was working with was not particularly large (@1.3 GB) with the majority of the data residing in just a few tables. (For reference this is a standalone Java HCP app). Since remote tunnel access to production HCP Databases is not available, the only way to get the raw data was through a HCP ticket, the support team were helpful and responsive, and after a couple of hours I had my exported dataset.

 

I loaded the data into my local MaxDB instance and chose to use the MaxDB loadercli command line tool to export the data out of the source DB and into CSV files, during this step I transformed some of the data as well put the data into the corresponding column order which matched my target table to make things simpler. I had one challenge with using loadcli which I posted to MaxDB SCN forums - but still no solution, so I had to clean some of the CSV files up using regex in my text editor.

 

Here is an example export statement from MaxDB. The use of CASE statements can be helpful to transform data types:

 

 

EXPORT     COLUMNS ID,     CREATIONDATE,     DESCRIPTION,     CASE WHEN TYPE = 'single' THEN '1' WHEN TYPE = 'group' THEN '2' WHEN TYPE = 'public' THEN '3' ELSE  ' ' END 
FROM     NEO_12345.T_TABLE
DATA OUTSTREAM 'T_TABLE.csv'

Once I had a set of CSV’s I needed to decide how I was going to load the data into my HANA instance, after some trial and error of using the HDBTI files, via the build in IMPORT statements (I have no way of getting access to the file system of the HANA instance), using HANA Studio, or a dedicated import script via XSJS - I decided on the latter. I found the studio process just did not work for some reason, but the xsjs import process was reasonably quick. Since I was going to do multiple loads, I thought it would be pretty helpful to have a tool which facilitates loading the CSV files and wrote a small open source tool called XS Import.

 

 

Overall the process was fairly straight forward, aside from some of the minor challenges described above

 

If you have done any similar migrations in HCP? and had a different experience, please feel free to share your thoughts on some of the processes and methods I chose above as I would like hear some of your opinions. (I am hoping to hear if there is faster/better/simpler?)

 

Thanks and cheers, Paul


XSImport - a HANA app for importing CSV's

$
0
0

 

 

During a recent small migration project from a HCP MaxDB DB to a HCP HANA DB I needed to import multiple CSV files, and go through a series of dry runs during the process. Since I was going to be doing the loads multiple times, I decided it would be helpful to write a small front end to make the processes of uploading failrly large CSV files into HANA simpler.

 

Over the past few months I have heard the question: "Whats the best way to get data into HANA from CSV?" While there are a few options if you have a HANA appliance and system level access to the system, there are not too many options with a HCP instance. After trying/testing the CSV import from Studio (size limit), a hdbti file (not great for multiple/repetitive options/file size challenges) it does not leave you with too many choices. After doing some digging it seems quite a few people have uploaded data through a server side script (XSJS) successfully and using the BATCH option while processing is pretty fast.

 

So I decided to put a front end together called XSImport - it provides a quick and easy way to load CSV files into HANA using a native XS application. You can find the project on Github here. It has a couple of helpful features like providing a preview of the data set, ability to delete all files already in the table, and a overview of the target table data types and dimensions. Some helpful features if you are going through a similar migration.

 

Below is a screenshot of the app, if anyone has any suggestions for improvement or would like to contribute to the project, please feel free.

 

A couple of you might point out and ask why I didn't upload the file directly to the HANA instance? Unfortunately if you are using a HCP HANA instance you don't have access to the file system. If you do, this method from Thomas is still a great option: Importing CSV files into SAP HANA - In-Memory Business Data Management - SCN Wiki

 

Features:

 

  • Preview data before uploading
  • Specify a CSV preview row
  • Overwrite option
  • Displays target table info (column names, types, sizes)
  • Uses BATCH for faster imports
  • Can be used for HCP HANA or dedicated HANA instances

 

Limitations:

 

  • Since files are uploaded via the browser, any browser size POST limitations come into effect
  • Network speed to HCP or HANA will effect the file upload speed

 

 

Distinct counter using counter option in SAP HANA

$
0
0

Author Bio


MP Reddy is a Principal Consultant at NTT DATA from the SAP Analytics Practice.


Summary

 

Business want to identify what is the count distinct products brought by each customer.

In this article let’s look at how to calculate distinct count for any attribute (column) in HANA with the help of one of the new feature available ‘Counter ‘.

 

Solution:

 

In this kind of requirement, just count of products would give us wrong results because customer might have brought same product multiple times. We should be able to calculate distinct count of products to get right solution.

 

 

This can be achieved with the help of ‘Counter’ feature available in both analytic and calculation view in SAP HANA.

 

Step #1:

 

To create new counter, right click on calculated column folder ->new counter.

 

Step #2:


Enter the technical name and description for new counter. We need to select on which column we would like to have distinct count. In our case, it is “PRODUCT CODE”.

 

 

Step #3:

 

Click OK to finish the creation process and see the new count column listed under ‘calculated column’ folder. To test our final results, we have also created an additional calculated measure which just does count of PRODUCT CODE.

 

 

Step #4:

 

Activate the analytical view and click ‘data preview’ to check the data. When we check the distinct products (green) for each customer, we can easily identify the difference count when compared to actual product count (blue).


 

      

Thus we have seen the process of creating counter in SAP HANA to achieve distinct count functionality.

 

 

Related Content

http://scn.sap.com/thread/3155627

https://scn.sap.com/thread/3151180

Hana Adapter SDK - Adapter capabilities

$
0
0

This post is part of an entire series

Hana Smart Data Integration - Overview

 

When designing the Data Provisioning Solution, one key element were the Adapter Capabilities. Initially the assumption of SDA was every adapter is database-like. So joins can be pushed down, every kind of filter etc. No problem if the source is a database which supports that, then the entire exercise is to translate the Hana SQL the adapter gets into the SQL syntax of the source database understands.

If the source is not a database, that would mean the adapter has to implement it. The adapter implementing a join manually? An aggregation? A substring function?? That would have been easy for the SDA team but make adapter development very difficult.

Instead the adapter does return a list of capabilities it supports at adapter, table and column level and everything else the adapter does not support is executed as a second step inside Hana.

 

From a technology point of view that is quite simple to do.

Example: The user executes a SQL "select column3, column5 from virtual_table where column2 = 99". This SQL is equivalent to a SQL in the form of "select column3, column5 from (select column2, column3, column5 from virtual table) where column2 = 99". Some operations are put into the inner SQL - all the ones the adapter supports - and the rest is done in the outer SQL, the one executed in Hana. The capabilities control what goes into the inner SQL versus what is outside. In this concrete example, the capabilities would tell that projects (=returning individual columns) is supported, a where clause is not.

 

Pushdown of filters - example

Obviously this topic of adapter capabilities can get very complex quickly. Twitter is a good example of a source which is certainly not a database. According the the API documentation it supports where clauses.

see: The Search API | Twitter Developers

At a first look the logic is straight forward

Search StringDescriptionSQL equivalent
watching nowcontaining both “watching” and “now”where Tweet like '%watching%' and Tweet like '%now%'
"happy hour"containing the exact phrase “happy hour”where Tweet like '%happy hour%'
love OR hatecontaining either “love” or “hate” (or both).where Tweet like '%love%' or Tweet like '%hate%'
beer -rootcontaining “beer” but not “root”where Tweet like '%beer%' and not Tweet like '%root%'

 

Filtering on dates is more of a challenge as

Search StringDescriptionSQL equivalent
since:2015-07-19all tweets with sent date since 2015-07-19where sent >= '2015.07.19'
until:2015-07-20all tweets with sent date until 2015-07-20where sent <= '2015.07.20'

 

Therefore a BETWEEN would work, an EQUAL also but a strict GREATER THAN cannot be pushed into the Twitter API.

 

Queries on long/lat are possible but only in the form of "near". No idea how to express that in pure SQL. And those are AND always, no way to have a query similar to "All Tweets where either the text contains Boston OR it is near Boston".

 

Pushdown - general guideline

 

The guideline for pushdowns has to be that the resulting dataset should be the same if the filter is pushed down to the source or applied in Hana. Although obvious, that could present problems. Is a Twitter search string "now" really the equivalent of "Tweet like '%now%'"? No, it is not. In SQL the like operator works on characters, the Twitter search on words. So the text "nowhere" would not match according to Twitter but would according to the database.

Other effects could be casing, like is 'NOW' = 'now'? Or trailing blanks 'now ' = 'now'?

When performing an order-by, what is the sort order of Hana compared to the source? A classic example are Western European characters 'Austria' < 'Österreich' = 'Oesterreich' < 'Russia'? In a binary order based on the UTF-8 character set it would be 'Austria' < 'Oesterreich' < 'Russia' < 'Österreich'.

 

As a consequence when I write an adapter I tend to pushdown rather less than more and if one kind of pushdown is really required, okay, then my adapter is not 100% consistent. Better inconsistent than useless because of performance reasons.

 

Adapter Capabilities

 

On the adapter level the global capabilities are set. When creating/refreshing an adapter within Hana, the Data Provisioning Server of Hana calls the getCapabilities() method which has to return a list of Capabilities. These Capabilities are then stored in the Hana Data Dictionary and control the various aspects of the Adapter.

The majority of the capabilities deal with the kind of statements the adapter supports, e.g. AdapterCapability.CAP_SELECT tells Hana that this adapter supports select queries. A CAP_INSERT would mean the adapter supports loading as well.

Other Capabilities deal with all the kind of Hana functions the Adapter is able to understand, e.g. CAP_BI_UPPER would allow Hana to even push an upper() function in the SQL sent to the adapter.

But not all is related to pushdown, e.g. CAP_METADATA_ATTRIBUTE tells Hana this adapter does wish to store additional table properties.

As said, all these Capabilities are global for this adapter.

In case it is not sufficient to have global capabilities only, there are the adapter Capabilities CAP_TABLE_CAP and CAP_COLUMN_CAP. With these the SQL optimizer of the federation layer does the extra steps of checking the table metadata and its capabilities being set on table level or even down to every column.

Of course it would be optimal to control every single of these settings on a table level and column level. The downside of such total flexibility would be the optimizer performance, the kind of logic the optimizer has to go through to create the execution plan. Therefor the majority of the capabilities can be controlled on adapter level only, important exceptions on table/column level.

 

Where-clause related Capabilities

 

The most important capabilities revolve around the pushdown of where clauses. If the amount of data is small, the absolute time difference between reading all rows and executing the filter in Hana versus returning just the matching data, is not that big. But in the majority of cases the amount should be filtered as early as possible, meaning the adapter should get the information about the filters.

To enable that the CAP_WHERE has to be set. Without, no where-clause will ever be pushed down into the adapter. Next question is the type of where clause the adapter supports. Examples with increasing complexity:

 

where clauseadditional Capabilities to be setComment
where value=1CAP_WHERE, CAP_SIMPLE_EXPR_IN_WHERE
where value > 2CAP_WHERE, CAP_SIMPLE_EXPR_IN_WHERE, CAP_NONEQUAL_COMPARISONincludes: >, >=, <, <=, <>
where value between 1 and 3CAP_WHERE, CAP_BETWEENNote: adapter will receive "value >= 1 and value <= 3" even if CAP_AND is turned off
where value in (1,2,3)CAP_WHERE, CAP_SIMPLE_EXPR_IN_WHERE, CAP_IN

Note: adapter will receive "(value = 1 or value = 2 or value = 3)"

includes: not in

where text like '%abc%'CAP_WHERE, CAP_SIMPLE_EXPR_IN_WHERE, CAP_LIKEincludes: not like
where value = 1 and text = 'abc'CAP_WHERE, CAP_SIMPLE_EXPR_IN_WHERE, CAP_AND_DIFFERENT_COLUMNS
where (value = 1 or value = 2) and text = 'abc'CAP_WHERE, CAP_SIMPLE_EXPR_IN_WHERE, CAP_AND_DIFFERENT_COLUMNS, CAP_OR

 

Table and Column level capabilities

 

On table level the capabilities available to control the pushdown are

  • CAP_TABLE_SELECT, .._INSERT, .._UPDATE, .._DELETE, .._UPSERT
  • CAP_TABLE_AND, .._OR, .._AND_DIFFERENT_COLUMNS, .._OR_DIFFERENT_COLUMNS

So the idea is that some tables support reading and loading, other tables just selecting data from it. And the combinations of AND/OR one table might support can be different to another.

Remember to set the AdapterCapability.CAP_TABLE_CAP in the adapter, else these settings are not even read. And the table level capabilities are part of the table metadata, so the TableMetadata object the method importMetadata() returns should contain the table capabilities list.

 

If there is the requirement to have different capabilities per column, the AdapterCapability.CAP_COLUMN_CAP is set on adapter level and/or the table level capability TableCapability.CAP_TABLE_COLUMN_CAP. The list of capabilities added to each Column of a table includes:

  • CAP_COLUMN_FILTER
  • CAP_COLUMN_LIKE, CAP_COLUMN_IN, CAP_COLUMN_NONEQUAL_COMPARISON, CAP_COLUMN_BETWEEN
  • CAP_COLUMN_GROUP
  • CAP_COLUMN_INNER_JOIN, CAP_COLUMN_OUTER_JOIN
  • CAP_COLUMN_SORT
  • CAP_COLUMN_SELECT, CAP_COLUMN_UPDATE, CAP_COLUMN_UPSERT, CAP_COLUMN_INSERT

Here the idea is that one column can be selected but no filter be applied on. One column is read only, the other supports updates. And some control of the complexity of the filters supported.

 

Gotchas

 

The assumption of the capabilities is that they are static. Set once by the adapter and then never changed again. This allows to keep the capability information in the Hana data dictionary tables, e.g. in the SYS.ADAPTERS_ table the column CAPABILITIES is a BIGINT with a bitmap mask of all set adapter capabilities.

As a consequence of that, whenever the capabilities do change, the alter-adapter command has to be executed to reread that information, e.g. using this syntax

alter adapter "CapabilitiesTestAdapter" refresh at location agent "mylaptop";

 

But even that is not enough, the optimizer has a few more places to cache the execution plans and the such. And if table/column capabilities are used, then the information is part of the Hana virtual table. Hence it is advised to drop and recreate the virtual tables as well.

 

Testing

 

In order to simplify playing with the adapter capabilities, a CapabilityTestAdapter has been created and is available in github.

hana-native-adapters/CapabilitiesTestAdapter

 

This adapter has two tables, the SQLTEXT table and the ADAPTERCAPS table. Using a statement like

update v_adaptercaps set isset = 'TRUE' where CAPABILITY = 'CAP_NONEQUAL_COMPARISON';

the individual adapter capabilities can be turned on and off.

 

Then the Hana dictionary is refreshed using commands similar to

alter adapter "CapabilitiesTestAdapter" refresh at location agent "mylaptop";

drop table v_sqltable;

create virtual table v_sqltable at "captest"."<NULL>"."<NULL>"."SQLTABLE";

 

And finally the pushdown can be tested by either executing the query or looking at the explain plan of each statement.

select * from v_sqltable where rownumber>=1;

 

Note: The Capability settings are kept in static memory, meaning whenever the adapter is restarted, it starts with the default settings. Also keep in mind that ADAPTERCAPS table returns the capabilities set in the adapter at that moment, not the Hana data dictionary information and what is actively used by the Hana optimizer currently.

Finally, a set of four capabilities cannot be turned off because else the update statement would not work:

  • AdapterCapability.CAP_SELECT
  • AdapterCapability.CAP_UPDATE
  • AdapterCapability.CAP_WHERE
  • AdapterCapability.CAP_SIMPLE_EXPR_IN_WHERE

Announcing CenSQL: A CLI client for SAP HANA

$
0
0

I created CenSQL to provide command history and shorthand commands useful for monitoring SAP HANA which aren't available in the hdbsql client.


CenSQL provides a lot more functionality which is not present in hdbsql, such as shortcuts for sql queries useful for any DB admin or developer for example \al for viewing open alerts on the instance you are connected to. I also put a very large amount of work into the usability of CenSQL, making it more like a modern CLI REPL tool.

CenSQL does not need to be installed on the HANA server, and can be installed on a different linux machine to then access the HANA Server remotely providing the correct port (3XX15, where XX is the instance No.) is open. CenSQL uses password based authentication for each connection.

For Installation Instructions see https://www.npmjs.com/package/censql#installation

Here are some examples of using CenSQL:

Viewing the current CPU usage per service:

Screenshot from 2015-09-18 11:54:21.png

Viewing a memory graph:

Screenshot from 2015-09-18 11:52:53.png

Help and command list

Screenshot from 2015-09-18 12:08:12.png

 

Finally, I made CenSQL for my own use initially and was told my colleagues it could be a good addition to the tools available for HANA developers and administrators. I look forward to getting feedback on how it could be improved in the future.

PlanViz - Improving SAP HANA Performance

$
0
0

SAP HANA enables organizations to optimize their business operations by analyzing large amounts of data in real time. The high performance platform can deliver stunning results only when it is used correctly by the programmers.

 

Performance is one of the most important aspect for a programmer during development but it is missed majority of the times. Programmers should follow the development guidelines to optimize the performance.

 

Software performance has 80/20 rule that 80% of the performance optimization is achieved by 20% of the code.

 

Now if any performance issue popups then you cannot find all the bottleneck by looking at the code or the SAP Hana modelling. You have to actually run the code to measure the performance. SAP ABAP has several tools like ST05 and SE30 which helps to measure performance.

 

SAP Hana has tools like as PlanViz which help to analyze the runtime performance queries and pinpoint bottlenecks in your system. You can remove those bottlenecks and increase performance

 

I hope you will find this guide helpful and I'll be glad to receive your feedbacks. The book contains step-by-step instructions and screenshots that will enable you to explore your PlanViz in SAP Hana. The book is available online on SAP Press Website

 

pic.PNG

 

Thanks & Regards,

Akash Kumar

Viewing all 676 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>