How to implement Loops in U-SQL - u-sql

Is is possible to implement Loops (while/for) in U-SQL without using C#. If no, can anyone share the c# syntax to implement loops in u-sql.
I am extracting files from a particular date to a date, but right now I am extracting this by writing file path manually.
DROP VIEW IF EXISTS dbo.ReadingConsolidated;
CREATE VIEW IF NOT EXISTS dbo.ReadingConsolidated
AS
EXTRACT
ControllerID int?,
sensorID int?,
MeasureDate DateTime,
Value float
FROM
"adl://datalake.azuredatalakestore.net/2015/7/1/Reading.csv",
"adl://datalake.azuredatalakestore.net/2015/7/2/Reading.csv",
"adl://datalake.azuredatalakestore.net/2015/7/3/Reading.csv",
"adl://datalake.azuredatalakestore.net/2015/7/4/Reading.csv",
"adl://datalake.azuredatalakestore.net/2015/7/5/Reading.csv",
"adl://datalake.azuredatalakestore.net/2015/7/6/Reading.csv",
"adl://datalake.azuredatalakestore.net/2015/7/7/Reading.csv"
Note: these files are present in different folder.
Can above thing is possible using loop?

The proper way to do this is to use virtual columns, then rely on partition elimination so that only files matching predicates will actually be read (you can confirm that in the job graph).
CREATE VIEW IF NOT EXISTS dbo.ReadingConsolidated
AS
EXTRACT
ControllerID int?,
ParameterID int?,
MeasureDate DateTime,
Value float,
date DateTime
FROM
"adl://datalake.azuredatalakestore.net/{date:yyyy}/{date:M}/{date:d}/Reading.csv";
#res =
SELECT * FROM dbo.ReadingConsolidated
WHERE date BETWEEN DateTime.Parse("2015/07/01") AND DateTime.Parse("2016/07/07");

Related

DateTime Type Gets Converted into bigint in SQLite

I Am new to SQLite.I am trying to create table from my model class in which I have DATETIME field.when sqlite table creatd from This Class The DATETIME column creted of type bigint.I know the reason because SQLite doesn't support DATETIME
So what should I do to create Column Of type TEXT in SQLITE without Chnging DATETIME type in my model Class. I found some post regarding to this but don't get satisfactory solution.please help.Thanks
You can configure how DateTimes are stored in your connection string. Change the attribute DateTimeFormat to either "ISO8601" or "CurrentCulture".
But: I would not recommend to do that. If you ever want to sort by a datetime or you want to filter rows (give me all entries from the last 2 weeks) then the bigint approach is the most efficient one. While ISO8601 datetime strings are sortable, that is most likely not the case with localised CurrentCulture strings. So if you really want human-readable strings in your database then choose the ISO version.

Whats the best way to query DynamoDB based on date range?

As part of migrating from SQL to DynamoDB I am trying to create a DynamoDB table. The UI allows users to search based on 4 attributes start date, end date, name of event and source of event.
The table has 6 attributes and the above four are subset of it with other attributes being priority and location. The query as described above makes it mandatory to search based on the above four values. whats the best way to store the information in DynamoDB that will help me in querying based on start date and end date fairly easy.
I thought of creating a GSI with hashkey as startdate, rangekey as end date and GSI on the rest two attributes ?
Inshort:
My table in DynamoDB will have 6 attributes
EventName, Location, StartDate, EndDate, Priority and source.
Query will have 4 mandatory attributes
StartDate, EndDate, Source and Event Name.
Thanks for the help.
You can use greater than/less than comparison operators as part of your query http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html
So you could try to build a table with schema:
(EventName (hashKey), "StartDate-EndDate" (sortKey), other attributes)
In this case the sort-key is basically a combination of start and end date allowing you to use >= (on the first part) and <= (on the second part)... dynamodb uses ASCII based alphabetical ordering... so lets assume your sortKey looks like the following: "73644-75223" you could use >= "73000-" AND <= "73000-76000" to get the given event.
Additionally, you could create a GSI on your table for each of your remaining attributes that need to be read via query. You then could project data into your index that you want to fetch with the query. In contrast to LSI, queries from GSI do not fetch attributes that are not projected. Be aware of the additional costs (read/write) involved by using GSI (and LSI)... and the additional memory required by data projections...
Hope it helps.

Fetching documents in xquery

I need to fetch documents from DB in xquery between dates [from date and to date].
From Date - 30 days before from Current Date
To Date - current date
In every document, I have an attribute named "loadDate". I have to fetch without creating an index for this attribute. Is that possible?
Please help.
Thanks,
-N
Assuming that your 'loadDate' attribute has type xs:date, and making up an imaginary structure for your documents, it sounds as if your query is simply:
/myns:doc
[#loadDate gt (current-date() - xs:dayTimeDuration('P30D'))]
Such a query might be slower without an index, but why would it not be possible? In a declarative query language, the general principle is that the existence of an index should not change the meaning of any query, only the speed with which it can be evaluated.

Sqlite C/C++ API - Get timestamp value with select query

I use sqlite3 C/C++ API to retrieve rows from a table using SELECT query. I don't see any sqlite3_column_timestamp() to retrieve a timestamp column value after sqlite3_step().. How to get timestamp values ?
SQLite does not have a special timestamp data type.
When you want to use any of SQLite's date and time functions, you have to store timestamps in one of the formats supported by them, i.e., a string like YYYY-MM-DD HH:MM:SS or HH:MM:SS, a julian date number, or a Unix timestamp number.
You can declare a table column type as DATETIME, but SQLite will just ignore that type; SQLite always allows to put values of any type in any column. Such a declaration would be useful only as documentation.
The column/value accessors will only have types corresponding to the data types they support directly (NULL, INTEGER, REAL, TEXT, BLOB).
You would use the TEXT access to get/set the column value of dates.
There are some helper functions within SQL that they provide that let you to handle them in your queries.
I am not familiar with SQLite Manager, but I would assume that it is only reporting the data type that the table was declared with.
When parsing CREATE statements, sqlite understands the intention of many well supported datatypes and automatically maps them to what is appropriate for its internal storage structure. VARCHAR would be mapped to TEXT, for instance. I assume the column was declared DATETIME and sqlite just internally mapped it to TEXT.

Creating a DateType field on sqlite3

i want to create a table on sqlite with one field as DateTime (YYYY-MM-DD) , how i can create it?
i'm trying with:
create table test (_date datetime);
but i'm not sure if the datatype is correct 'cause i can do this:
create table test (_date nyanType);
and no error occours
SQLite is rather unique in that its columns are not statically typed. You can technically store a string in a column that was created as an integer column.
If you check out the SQlite Documentation for types, you'll see that SQLite dosn't have a date type, but it exposes date and time functions that are suitable for manipulating dates that are stored as TEXT, REAL or INTEGER. You should use those instead.

Resources