I have a biztalk 2013-r2 schema for a txt file, that sometimes does not have a value for the last column.
With value:
Name, address, city, state, zip, phone
Mike Myers, 30 West Street, NY, 10001, 2124444444
Without value:
Name, address, city, state, zip, phone
Mike Myers, 30 West Street, NY, 10001,
When the column has value, there are no issues.
When the last column has no value, I get the following error:
PublicKeyToken=92c4a30afea05f3e" Source: "Flat file disassembler" Receive Port: "rp_List" URI: "C:\BiztalkTestFiles\List\myfiles\*.txt" Reason: Unexpected data found while looking for:
'\r\n'
The current definition being parsed is Mydefinition. The stream offset where the error occured is 101. The line number where the error occured is 2. The column where the error occured is 6.
Is there a workaround for this type of issue?
Schema attached:
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://usgeSales.ldcSyncList.schemas.MySchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://usgeSales.ldcSyncList.schemas.MySchema" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
<b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="MySchema" />
</xs:appinfo>
</xs:annotation>
<xs:element name="MySchema">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element maxOccurs="unbounded" name="MySchema_Detail">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Name" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Address" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="State" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Zip" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="4" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element maxOccurs="0" name="Phone" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="5" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
file 1: with.txt
Mike Myers, 30 West Street, NY, 10001, 2124444444
file 2: without.txt
Mike Myers, 30 West Street, NY, 10001,
Usually you simply set the Min Occurs on the last element to 0
Also make sure that your records do actually finish with a CR & LF (use NotePad++ and show All Symbols)
The fact that it is complaining on line 2 column 11, your problem may actually be something else.
Related
I have this CSV input:
Username,"Certification ID number","Certification Name","Date completed","Date due"
nlxxxxxx,NLaaa,"NL - Test certificaat",1/12/2021,"15/12/2022, 07:55"
nlyyyyyy,CPbbb-cert,"Test Law certificaat",1/12/2021,"30/11/2023, 09:10"
I get from BizTalk Test Mapper the following error:
error btm1041: Native Parsing Error: Unmatched wrap character "
The current definition being parsed is Certificate. The stream offset where the error occured is 153. The line number where the error occured is 2. The column where the error occured is 68.
I assume the error comes from the Date due field. I setted this field with wrap character property (see XSD below).
<?xml version="1.0" encoding="utf-16" ?>
<xs:schema xmlns="http://Securitas.ESB.HR.LMSToAfas_ManageCertificate.Schemas.LMSCertificate" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Securitas.ESB.HR.LMSToAfas_ManageCertificate.Schemas.LMSCertificate" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
<b:schemaInfo standard="Flat File" codepage="65001" default_pad_char="" pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="LMSCertificate" />
</xs:appinfo>
</xs:annotation>
<xs:element name="LMSCertificate">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xA" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Header">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Username" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationID" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationName" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateCompleted" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="4" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateDue" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="5" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element maxOccurs="unbounded" name="Certificate">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Username" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationID" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationName" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" wrap_char_type="char" wrap_char=""" pad_char_type="hex" pad_char="0x20" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateCompleted" type="xs:date">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="4" pad_char_type="hex" pad_char="0x20" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateDue" type="xs:dateTime">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="5" wrap_char_type="char" wrap_char=""" pad_char_type="none" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
I can't figure out where I am wrong
First of you need to check what your lines end with, you've defined it as LF, but if it is CR LF it will throw the error above. Change it to 0x0D 0x0A on the root node.
If you then get the below, then the final line doesn't have the delimiter and you need to change the child order on the root node LMS Certificate to Infix.
The current definition being parsed is Certificate. The stream offset where the error occured is 227. The line number where the error occured is 3. The column where the error occured is 72.
Next you will get the below errors
The 'DateCompleted' element is invalid - The value '1/12/2021' is invalid according to its datatype 'http://www.w3.org/2001/XMLSchema:date' - The string '1/12/2021' is not a valid Date value.
The 'DateDue' element is invalid - The value '15/12/2022, 07:55' is invalid according to its datatype 'DateTime' - The string '15/12/2022, 07:55' is not a valid DateTime value.
The 'DateCompleted' element is invalid - The value '1/12/2021' is invalid according to its datatype 'http://www.w3.org/2001/XMLSchema:date' - The string '1/12/2021' is not a valid Date value.
The 'DateDue' element is invalid - The value '30/11/2023, 09:10' is invalid according to its datatype 'DateTime' - The string '30/11/2023, 09:10' is not a valid DateTime value.
Set the DateCompleted Custom Date/Time Format to d/MM/yyyy, that will eliminate two of the errors.
Now the DueDate one is tricky, I tried a Custom Date/Time Format of d/MM/yyyy, hh:mm, but it didn't work and stil threw the same two errors. If I changed it to a String it parsed correctly. A clue was when I generated an native instance was that it would populate it as DueDate. After I copied your other date and renamed it, updated it to a DateTime and updated the custom format it worked.
Differences
child_delimiter="0x0D 0x0A" on LMSCertificate
Not sure if this got corrupted when you posted to StackOveflow, but wrap_char=""" isn't valid, it should be wrap_char="""
datetime_format="d/MM/yyyy" on DateCompleted
pad_char_type="hex" pad_char="0x20" datetime_format="d/MM/yyyy, hh:mm" on DueDate
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://Securitas.ESB.HR.LMSToAfas_ManageCertificate.Schemas.LMSCertificate" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Securitas.ESB.HR.LMSToAfas_ManageCertificate.Schemas.LMSCertificate" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
<b:schemaInfo standard="Flat File" codepage="65001" pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="LMSCertificate" />
</xs:appinfo>
</xs:annotation>
<xs:element name="LMSCertificate">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0x0D 0x0A" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Header">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Username" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationID" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationName" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateCompleted" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="4" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateDue" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="5" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element maxOccurs="unbounded" name="Certificate">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Username" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationID" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CertificationName" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" wrap_char_type="char" pad_char_type="hex" pad_char="0x20" wrap_char=""" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateCompleted" type="xs:date">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="4" pad_char_type="hex" pad_char="0x20" datetime_format="d/MM/yyyy" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DateDue" type="xs:dateTime">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" pad_char_type="hex" pad_char="0x20" datetime_format="d/MM/yyyy, hh:mm" sequence_number="5" wrap_char_type="char" wrap_char=""" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
Output
<LMSCertificate xmlns="http://Securitas.ESB.HR.LMSToAfas_ManageCertificate.Schemas.LMSCertificate">
<Header xmlns="">
<Username>Username</Username>
<CertificationID>"Certification ID number"</CertificationID>
<CertificationName>"Certification Name"</CertificationName>
<DateCompleted>"Date completed"</DateCompleted>
<DateDue>"Date due"</DateDue>
</Header>
<Certificate xmlns="">
<Username>nlxxxxxx</Username>
<CertificationID>NLaaa</CertificationID>
<CertificationName>NL - Test certificaat</CertificationName>
<DateCompleted>2021-12-01</DateCompleted>
<DateDue>2022-12-15T07:55:00.0000000</DateDue>
</Certificate>
<Certificate xmlns="">
<Username>nlyyyyyy</Username>
<CertificationID>CPbbb-cert</CertificationID>
<CertificationName>Test Law certificaat</CertificationName>
<DateCompleted>2021-12-01</DateCompleted>
<DateDue>2023-11-30T09:10:00.0000000</DateDue>
</Certificate>
</LMSCertificate>
P.S. I also assumed that the extra lines in your sample were due to trying to format it for StackOverflow, and that it actually looks like this
Username,"Certification ID number","Certification Name","Date completed","Date due"
nlxxxxxx,NLaaa,"NL - Test certificaat",1/12/2021,"15/12/2022, 07:55"
nlyyyyyy,CPbbb-cert,"Test Law certificaat",1/12/2021,"30/11/2023, 09:10"
My BizTalk orchestration imports only 1 row.
I cannot figure out what is wrong, since there are no error messages or warnings.
I am using:
WCF
Orchestration
Receive Pipeline
Stored Procedure
CSV Flat File
I think the issue is in the csv schema.
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://ElecFFILEImport.FFILECSVSchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://ElecFFILEImport.FFILECSVSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
<b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="true" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="FFILERoot" />
</xs:appinfo>
</xs:annotation>
<xs:element name="FFILERoot">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Header">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Header_Row" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element maxOccurs="unbounded" name="Detail">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
</xs:appinfo>
</xs:annotation>
<xs:element name="FFILECompany" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="1" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="AccountName" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="HomeAddress" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="AccountNumber" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="4" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="ServiceNumber" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="5" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="BNumber" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="6" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="DS" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="7" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="LPC" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="8" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="RDate" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="9" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="PDate" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="10" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="BOption" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="11" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="RSCode" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="12" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="CPAssigment" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="13" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Status" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="14" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
But just in case here is a sample file (fake data)
FFILECompany,Account Name,Home Address,Account Number,Service Number,Bill Number,Delivery Service,LPC,RDate,PDate,BOption,RSCode,CPAssignment,Status
Company A,NORM W MYERS,0 X WEST STREET STRONGHURST IL 61480,6996293052,439998937,5,DS1,DSHDHD-HP,2014-01-02,,BILL,,,ACTIVE
Company B,ROSABEL BLESKIN,982001 RUSTIC ST ROBINSON IL 62466,1499936152,53875283,13,DS1,RDDD-HP,2014-10-10,,BILL,,,ACTIVE
Company C,TIEE E DISNEY,140505 BEL AIRE DR BELLEVILLE IL 62299,1058999074,40564211,10,DS1,RESSS-HP,2015-01-11,2015-11-09,BILL,,,PENDING INACTIVE
Company D,STEVE BROWN,1044920 COLLEGE AVE ALTON IL 62154,1048999118,54999928,2,DS1,RSS-HP,2014-06-26,,BILL,,,ACTIVE
And finally, here is my stored procedure:
As you can see I only import a few columns:
CREATE PROCEDURE [dbo].[AccountImport]
#Commodity NVARCHAR(255) , -- Hard Coded value inside the map
#LDCCode NVARCHAR(2) , -- Hard Coded value inside the map
#AccountNumber NVARCHAR(255) ,
#AccountName NVARCHAR(255) ,
#HomeAddress NVARCHAR(255)
AS
INSERT INTO dbo.[CustomerTable]
( Commodity ,
LDCCode ,
AccountNumber ,
AccountName ,
HomeAddress
)
SELECT #Commodity ,
#LDCCode ,
#AccountNumber ,
#AccountName ,
#HomeAddress
DECLARE #MID AS INT
SELECT #MID = ##IDENTITY
--This stored procedure will parse the address and separate Address city and state using a space
EXEC [sync].[ParseAddress] #ID = #MID
Again, there are no errors, and one row is imported every time (the first row).
Make sure you set up your schema as an envelope schema to cater for multiple records. See the below links
https://msdn.microsoft.com/en-us/library/aa546772.aspx
http://www.codeproject.com/Articles/507336/Envelope-Schema-and-Debatching
How does the message that enter the orchestration look like? If it is only one row, then the problem is in receive pipeline; solution for that:
I think you should define two separate schemas, one for the header of the fileand one for the body of the csv file. Fill inf Flat File Disassembler pipeline like this:
DocumentSchema: BodySchemaName, AssemblyName
HeaderSchema: HeaderName, AssemblyName
If the message in the orchestration is full, without header, only body, then the problem is in the mapping, so you should provide us with more information. What does the schema you map into look like?
Make sure the "Max occurs" property on the record node of the inbound schema is set to "unbounded"
I have a .txt file which is in the similar format as below
1115151651515950000055 00012913702613000000000003000 139C0000007000000
1215151651121510000054 00022913803603000000000009000 000279A0000009000
1315115950000065516515 00032813104643000000000007000 000399B0000003000
121515160003290003290000010000000003000
The first 3 lines are body elements but the number of lines in the body part will be unknown(may occur from 1 to unbounded). There is no tag identifier in body part. The last line in the file is always a trailer.The trailer from the file is to be removed prior to parsing so that only the records need parsed. How can this be done using a Pipeline Component in the Flat File Disassembler.
You should not need to remove the trailer.
What you need it to define a Flat File Schema where the Body Record can occur unbounded and a separate record for the trailer.
You have to set the the delimiters on the Root in the example below, Child Delimiter = 0x0D 0x0A (CR LF), Child Delimiter Type = Hexadecimal, Child Order=Infix, but that may vary and you have to declare the correct delimiter and where the occur. For the above file I assumed that there was no CR LF after the trailer, hence chose Infix (delimiter occurs between) rather than Postfix (delimiter occurs after) if the last line does have a CR LF.
You can then define the Body and Trailer Structure to be either Delimited or Positional.
Update: To not have the trailer in the message to be processed, have a map on the receive port that only maps the Body records and not the trailer.
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://Scratch.FlatFile" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Scratch.FlatFile" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<b:schemaInfo standard="Flat File" root_reference="Root" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" />
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Root">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="1" child_order="infix" child_delimiter_type="hex" child_delimiter="0x0D 0x0A" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element maxOccurs="unbounded" name="Body">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="1" structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="BodyContents" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Trailer">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="2" structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="TrailerContents" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
I have a two types of flat file lines,
start with #, e.g. #111 222 333
start with #, e.g. #777 888 999
We can use tag identifier to identify these two types of lines. Unfortunately, these two lines may appear in any sequence in a file, like #####, or #####, or ##### etc.
If use flat file dissembler schema, seems the sequence must be in order, say #####, ###, it can disassemble all flat file content into XML format.
But if the file format is ###### or ######, it cannot dissemble all file content and will stop at the middle, say ###### can only disassemble ###, ###### can only disassemble ####.
I think it is because the flat file schema specify the sequence is # and then #, therefore, # cannot appear after # (please inform if my understanding is wrong).
Any solution to our case?
Thanks.
Put a the tagged records under a Choice Group that can occur multiple times.
See example below.
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://Scratch.FlatFileSchemaRandomTagOrder" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Scratch.FlatFileSchemaRandomTagOrder" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<b:schemaInfo standard="Flat File" root_reference="Root" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" />
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Root">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="1" child_order="postfix" child_delimiter_type="hex" child_delimiter="0x0D 0x0A" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="1" />
</xs:appinfo>
</xs:annotation>
<xs:element name="RecordHash">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="1" tag_name="#" child_order="infix" child_delimiter_type="char" child_delimiter=" " />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Field1" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Field2" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Field3" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="RecordAt">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="2" structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" tag_name="#" child_order="infix" child_delimiter_type="char" child_delimiter=" " />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Field1" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Field2" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="Field3" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo justification="left" sequence_number="3" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
Sample file
#111 222 333
#777 888 999
#111 222 333
Output
<Root xmlns="http://Scratch.FlatFileSchemaRandomTagOrder">
<RecordAt xmlns="">
<Field1>111</Field1>
<Field2>222</Field2>
<Field3>333</Field3>
</RecordAt>
<RecordHash xmlns="">
<Field1>777</Field1>
<Field2>888</Field2>
<Field3>999</Field3>
</RecordHash>
<RecordAt xmlns="">
<Field1>111</Field1>
<Field2>222</Field2>
<Field3>333</Field3>
</RecordAt>
</Root>
I have several config files that may or may not have certain attributes.
For example:
Property1 : value1
Property2 : value2
.
junk text
junk text : line count between useful data varies in every file.
junk text
.
Property3 : value3 //optional
Is there a way to only retrieve the properties by Tag Identifier only, and skip all the junk in between?
So far, I cannot figure out how to make skip the junk. "Ignore" is only allowed after "Record Sequence"
Yes, something like the below, where you tell it to look for certain tags, and have records without tags to consume those you aren't interested in.
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://Scratch.Junk" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Scratch.Junk" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<b:schemaInfo standard="Flat File" root_reference="Root" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="complexity" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" />
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Root">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="1" child_order="postfix" child_delimiter_type="hex" child_delimiter="0x0D 0x0A" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Property1">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="1" structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" tag_name="Property1 :" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="value1" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Property2">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" tag_name="Property2 :" sequence_number="2" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="value2" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="unbounded" name="Junk">
<xs:annotation>
<xs:appinfo>
<b:recordInfo sequence_number="3" structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="JunkValue" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" name="Property3">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" tag_name="Property3 :" sequence_number="4" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="value3" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element minOccurs="0" maxOccurs="unbounded" name="MoreJunk">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="5" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="JunkValue" type="xs:string">
<xs:annotation>
<xs:appinfo>
<b:fieldInfo sequence_number="1" justification="left" />
</xs:appinfo>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>