I am trying to use google datastore for my non GAE application.
For that i have created kinds and ancestor related entities in datastore using gcloud python library.
Also updated datastore index configuration for all the kinds using gcd tool via WEB-INF/datastore-indexes.xml file and its status' are serving.
However i can not successfully query the index based columns either in console or using gcloud lib.
Here is the query & traceback
from gcloud import datastore
ds = datastore.Client(dataset_id='XXXXXX')
query = datastore.Query(ds, kind='event')
query.add_filter('EvtName', '=', 'buy')
query.add_filter('EventDateTime', '<=', datetime.datetime(2015, 10, 22, 8, 45))
for itm in query.fetch():
print(dict(itm))
gcloud.exceptions.PreconditionFailed: 412 no matching index found.
here is my datastore-indexes.xml config
<?xml version="1.0" encoding="utf-8"?>
<datastore-indexes
autoGenerate="false">
<datastore-index kind="event" ancestor="true">
<property name="EvtName" direction="desc" />
<property name="EventDateTime" direction="desc" />
</datastore-index>
<datastore-index kind="att" ancestor="true">
<property name="EvtAttName" direction="desc" />
<property name="EventDateTime" direction="desc" />
</datastore-index>
<datastore-index kind="att_val" ancestor="true">
<property name="AttValue" direction="desc" />
<property name="EventDateTime" direction="desc" />
</datastore-index>
<datastore-index kind="user" ancestor="true">
<property name="EventDateTime" direction="desc" />
</datastore-index>
</datastore-indexes>
am i missing something?
All of your indexes are designed to be used with ancestor queries (note the ancestor=true). However, your actual query does not query within a specific ancestor.
In order to answer your specific query, you need the index:
<datastore-index kind="event" ancestor="false">
<property name="EvtName" direction="desc" />
<property name="EventDateTime" direction="desc" />
</datastore-index>
Or, if you actually do want to query for entities with a specific parent, make sure to add an ancestor filter with Query#hasAncestor(Key parentKey).
Related
I have created a REST - data service in WSO2EI an tried accessing through postman client tool, but I am getting this error.
End Point: http://LAPTOP-T4F1HOAM:8280/services/getStudRecNo?user_id=test8
The endpoint reference (EPR) for the Operation not found is /services/getStudRecNo?user_id=test8 and the WSA Action = null. If this EPR was previously reachable, please contact the server administrator.
Can anyone please help.
Thanks
dss code.
<data name="getStudRecNo" transports="http https local">
<description>get student unique id</description>
<config enableOData="false" id="mySchoolDB">
<property name="driverClassName">org.postgresql.Driver</property>
<property name="url">jdbc:postgresql://localhost:5432/mySchool</property>
<property name="username">admin</property>
<property name="password">admin</property>
</config>
<query id="GetStudentRecordNo" useConfig="mySchoolDB">
<sql>SELECT getstudentid FROM getstudentid(?)</sql>
<result outputType="json">{"entries": {"entry": [ { "getstudentid": "$getstudentid"} ]}}</result>
<param defaultValue="TEST" name="user_id" sqlType="STRING"/>
</query>
<operation name="getsrno">
<call-query href="GetStudentRecordNo">
<with-param name="user_id" query-param="user_id"/>
</call-query>
</operation>
<resource method="GET" path="getsrno">
<call-query href="GetStudentRecordNo">
<with-param name="user_id" query-param="user_id"/>
</call-query>
</resource>
</data>
You have to append the resource path ("getsrno" in your case) to the URL. Then the URL would look like,
http://LAPTOP-T4F1HOAM:8280/services/getStudRecNo/getsrno?user_id=test8
Further, since the resource's HTTP method is GET, the request should also be a GET request.
i'm setting up a ConcurrentMessageListenerContainer
<bean class="org.springframework.kafka.listener.ConcurrentMessageListenerContainer" id="messageListenerContainer">
<constructor-arg index="0" ref="consumerFactory"/>
<constructor-arg index="1" ref="containerProperties"/>
<property name="concurrency" value="2"/>
</bean>
ConsumerFactory use this config:
<util:map id="consumerConfig" map-class="java.util.HashMap">
<entry key="#{T(org.apache.kafka.clients.consumer.ConsumerConfig).BOOTSTRAP_SERVERS_CONFIG}"
value="${rp.kafka.bootstrap.servers}"/>
<entry key="#{T(org.apache.kafka.clients.consumer.ConsumerConfig).KEY_DESERIALIZER_CLASS_CONFIG}"
value="org.apache.kafka.common.serialization.StringDeserializer"/>
<entry key="#{T(org.apache.kafka.clients.consumer.ConsumerConfig).VALUE_DESERIALIZER_CLASS_CONFIG}"
value="org.springframework.kafka.support.serializer.JsonDeserializer"/>
<entry key="#{T(org.springframework.kafka.support.serializer.JsonDeserializer).TRUSTED_PACKAGES}"
value="*"/>
<entry key="#{T(org.apache.kafka.clients.consumer.ConsumerConfig).PARTITION_ASSIGNMENT_STRATEGY_CONFIG}"
value="org.apache.kafka.clients.consumer.RoundRobinAssignor"/>
<entry key="#{T(org.apache.kafka.clients.consumer.ConsumerConfig).ENABLE_AUTO_COMMIT_CONFIG}"
value="false"/>
</util:map>
and ContainerProperties are
<bean class="org.springframework.kafka.listener.ContainerProperties" id="containerProperties">
<constructor-arg>
<list>
<value>sendSMS</value>
</list>
</constructor-arg>
<property name="groupId" value="main"/>
<property name="messageListener" ref="messageListener"/>
<property name="ackMode" value="RECORD"/>
</bean>
My topic "sendSMS" has 5 partitions on 3-noded cluster with rep factor of 3, so i expect that each KafkaMessageListenerContainer created by Concurrent one (total 2 in that case) will take it's portion of partitions to handle. Hovewer, after an application is started i see in my debugger window that each listener is handling all 5! partitions
https://gyazo.com/183626ff60061b471858f8cc52573353
and message from 4-th partition (its where i have a message that hangs the processing and not being commited after restarts, but its not related to this issue) on the same offset is being delivered 2 times in different threads with different consumers! Why it happens so? Is it a bug or expected behavior?
You are not showing enough information. The concurrent container aggregates the assigned partitions for the child KafkaListenerContainers (one for each concurrency).
#Override
public Collection<TopicPartition> getAssignedPartitions() {
return this.containers.stream()
.map(KafkaMessageListenerContainer::getAssignedPartitions)
.filter(Objects::nonNull)
.flatMap(Collection::stream)
.collect(Collectors.toList());
}
You need to show logs for the re-delivery; turn on DEBUG logging for more information.
I have a bean as follows:
<bean id="myBean" class="MyBeanClass">
<constructor-arg value="\WEB-INF\myfile.dat"/>
</bean>
In the bean's contructor, I need to build the file's full path. To do that, I have to first find the app's root path first.
Thanks and regards.
Update
Per Michael-O's suggestion, here is my solution (so easy).
Spring bean:
<bean id="myBean" class="MyBeanClass">
<constructor-arg value="/myfile.dat"/> <!--under WEB-INF/classes-->
</bean>
Java:
public MyBeanClass(String path) throws Exception {
ClassPathResource file = new ClassPathResource(path);
lookup = new LookupService(file.getFile().getPath(), LookupService.GEOIP_MEMORY_CACHE);
}
Michael, thanks!!!
Use Spring's Resource class in your bean and spring will do the rest for you.
After seeing #curious1's edit, there is a better solution to his answer. Please do not use that. Go with this one:
beans.xml:
<!-- START: Improvement 2 -->
<context:annotation-config />
<bean id="service" class="LookupService">
<constructor-arg value="classpath:/myfile.dat"/> <!--under WEB-INF/classes-->
<constructor-arg>
<util:constant static-field="LookupService.GEOIP_MEMORY_CACHE"/>
</constructor-arg>
</bean>
<!-- END: Improvement 2 -->
<!-- Spring autowires here -->
<bean id="myBean" class="MyBeanClass" />
<!-- START: Improvement 1 -->
<bean id="myBean" class="MyBeanClass" />
<constructor-arg value="classpath:/myfile.dat"/> <!--under WEB-INF/classes-->
</bean>
<!-- END: Improvement 1 -->
Java:
public MyBeanClass(Resource path) throws Exception {
lookup = new LookupService(path.getInputStream(), LookupService.GEOIP_MEMORY_CACHE);
}
This is source-agnostic, does not rely on files and is the Spring way.
Edit 2: Rethinking my code, it can be even better:
public class MyBeanClass {
#Autowired
LookupService service;
}
and configure LookupService in your beans.xml.
Maybe you should consider using:
getClass().getClassLoader().getResourceAsStream()
inside constructor. This will use your classpath, so you "WEB-INF\myfile.dat", will be visible. Next think is use resource directory to put all resources in one directory (default: under root directory in WAR file)
I've inherited a project and am trying to get a set of integration tests running against an in-memory h2 database. In order for them to pass some tables, relationships and reference data needs creating.
I can see the problem in that the script referenced in RUNSCRIPT is being executed multiple times and therefore generating Index "XXX_IDX" already exists errors and other violations. So is there a way to force the script to only be run once or do I need a external database? It seems that the script is run on every connection which I assume is by design.
properties file
my.datasource.url=jdbc:h2:mem:my_db;DB_CLOSE_DELAY=-1;MODE=Oracle;MVCC=TRUE;INIT=RUNSCRIPT FROM 'classpath:/create-tables-and-ref-data.sql'
XML config
<bean id="myDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="${my.datasource.url}"/>
<!-- other properties for username, password etc... -->
</bean>
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="myDataSource"/>
</bean>
<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="myDataSource"/>
</bean>
<tx:annotation-driven transaction-manager="transactionManager"/>
many Java classes in the following pattern
#Component
public class SomethingDAOImpl implements SomethingDAO {
#Autowired
public SomethingDAOImpl(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
}
#Component
public class SomethingElseDAOImpl implements SomethingElseDAO {
#Autowired
public SomethingElseDAOImpl(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
}
With the default bean scope being singleton I thought this would just work, but I guess i'm missing something. Also, if I switch to an real Oracle instance that already has the tables and reference data set-up, the tests all pass.
In many cases, it is possible to write the SQL script so that no exceptions are thrown:
create table if not exists test(id int, name varchar(255));
create index if not exists test_idx on test(name);
I ended up using an alternative approach, as I could not write the SQL in a way that could be reapplied without error.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-3.2.xsd">
<jdbc:initialize-database data-source="myDataSource" enabled="true" ignore-failures="ALL">
<jdbc:script location="classpath:create-and-alter-tables-first-then-add-test-data.sql" />
</jdbc:initialize-database>
</beans>
which is executed once at context initialization.
Note: other namespaces and beans omitted for brevity.
I've got a content type based on ATFolder:
ConceptSheetFolderSchema = folder.ATFolderSchema.copy()
ConceptSheetFolderSchema['title'].widget.label = _(u"Title")
ConceptSheetFolderSchema['title'].widget.description = _(u"")
ConceptSheetFolderSchema['title'].storage = atapi.AnnotationStorage()
ConceptSheetFolderSchema['description'].widget.label = _(u"Description")
ConceptSheetFolderSchema['description'].widget.description = _("")
ConceptSheetFolderSchema['description'].storage = atapi.AnnotationStorage()
finalizeATCTSchema(ConceptSheetFolderSchema, folderish=True, moveDiscussion=False)
class ConceptSheetFolder(folder.ATFolder):
"""
This is the central container for concept sheets in the site
"""
implements(IConceptSheetFolder)
portal_type = "Concept Sheet Folder"
_at_rename_after_creation = True
schema = ConceptSheetFolderSchema
title = atapi.ATFieldProperty('title')
description = atapi.ATFieldProperty('description')
atapi.registerType(ConceptSheetFolder, PROJECTNAME)
I can add a ConceptSheetFolder no problem through the Plone interface, but I can't get this basic test to work:
class TestContent(unittest.TestCase):
layer = PROJECT_CONCEPTSHEETS_INTEGRATION_TESTING
def test_hierarchy(self):
portal = self.layer['portal']
# Ensure that we can create the various content types without error
setRoles(portal, TEST_USER_ID, ('Manager',))
portal.invokeFactory('Concept Sheet Folder', 'csf1', title=u"Concept Sheet folder")
portal['csf1'].invokeFactory('project.ConceptSheet', 'cs1', title=u"ConceptSheet")
portal['csf1']['cs1'].invokeFactory('project.ConceptMilestone', 'cs1', title=u"Approved")`
I get a error
Unauthorized: Cannot create Concept Sheet Folder when I try this test. I Googled around a bit and found this Nabble post, leading me to look at isConstructionAllowed() in Plone/CMFCore/TestTools.py. Using pdb, I found that ._queryFactoryMethod(), when run in this context, is returning 'None'.
So it appears the FactoryTool for this type isn't working, at least not in the test. I've got the test in the normal GenericSetup place (types.xml, Concept_Sheet_Folder.xml, factorytool.xml), and I'm at a lost as to what else could be causing this problem. Any ideas?
Bonus question: why does this work in the Plone interface but not in the test?
Edit (Dec 13, 2011): Here's my Concept_Sheet_Folder.xml
<?xml version="1.0"?>
<object name="Concept Sheet Folder"
meta_type="Factory-based Type Information with dynamic views"
i18n:domain="iedea.conceptsheets" xmlns:i18n="http://xml.zope.org/namespaces/i18n">
<property name="title" i18n:translate="">Concept Sheet Folder</property>
<property name="description"
i18n:translate="">A folder which can contain concept sheets.</property>
<property name="content_icon">++resource++conceptsheetfolder_icon.gif</property>
<property name="content_meta_type">Concept Sheet Folder</property>
<property name="product">iedea.conceptsheets</property>
<property name="factory">addConceptSheetFolder</property>
<property name="immediate_view">atct_edit</property>
<property name="global_allow">True</property>
<property name="filter_content_types">True</property>
<property name="allowed_content_types">
<element value="Concept Sheet" />
</property>
<property name="allow_discussion">False</property>
<property name="default_view">view</property>
<property name="view_methods">
<element value="view"/>
</property>
<alias from="(Default)" to="(dynamic view)"/>
<alias from="edit" to="atct_edit"/>
<alias from="sharing" to="##sharing"/>
<alias from="view" to="(selected layout)"/>
<action title="View" action_id="view" category="object" condition_expr=""
url_expr="string:${folder_url}/" visible="True">
<permission value="View"/>
</action>
<action title="Edit" action_id="edit" category="object" condition_expr=""
url_expr="string:${object_url}/edit" visible="True">
<permission value="Modify portal content"/>
</action>
</object>
I've run into this problem myself. The problem is that the your Archetype's factory is not yet properly registered by the time you are trying to create it.
That's why _queryFactoryMethod() returns None, as you found out.
The solution differs a bit on whether you are using Products.ZopeTestCase or the newer plone.app.testing as your testing framework.
However, in both cases you need to make sure that the add-on product that defines the Archetype (ConceptSheetFolder) that you are trying to create (via invokeFactory), has aready been installed.
When using Products.ZopeTestCase:
In the case that you are using Products.ZopeTestCase (and Products.PloneTestCase), you need to call
Products.ZopeTestCase.installProduct
You need to make sure that your installProduct call does not get deferred until after your test is called.
In Plone 4 this means that your installProduct call should not be in an #onsetup decorated function (although this will still work in Plone 3).
This mailing list discussion might further clear things up:
http://plone.293351.n2.nabble.com/invokeFactory-failing-on-Plone-4-PTC-but-working-on-Plone-3-td5755482.html
When using plone.app.testing:
If you are using plone.app.testing, you should call:
plone.testing.z2.installProduct
This should be done in the setUpZope method that you override from the PloneSandboxLayer.
For more info, read the description under setUpZope in plone.app.testing.helpers.py (Line 257)
https://github.com/plone/plone.app.testing/blob/2ef789f8173c695179b043fd4634e0bdb6567511/plone/app/testing/helpers.py