What is an "array of arrays"? - Arduino - arduino

I am following a tutorial about how to make a circuit involving a 4x4 keypad, multiple times a certain "vector array", you might say, has arisen that I do not know the meaning of.
I have searched around, but I have not found anything. Even on the Arduino reference sight.
Here is a code example from the video (https://www.youtube.com/watch?v=GyprsoB887Y&t=437s).
// "rows" and "cols" being defined as 4 before, so the line is essentially equal to char keys[4][4]
char keys[rows][cols] = {
{'1', '2', '3', 'A'},
{'4', '5', '6', 'B'},
{'7', '8', '9', 'C'},
{'*', '0', '#', '0'}
};
So, what is happening, specifically, it seems like an array of arrays. I see why it would have to be, it couldn't be one variable assigned to multiple arrays, it has to be a containing array that holds them. But I do see how it is declared as 4 by 4, but shouldn't it be 1 by 4? Also, what does keys[rows][columns] mean?
Couldn't it just be:
char keys[] = {
{'1', '2', '3', 'A'},
{'4', '5', '6', 'B'},
{'7', '8', '9', 'C'},
{'*', '0', '#', '0'}
};
Why isn't it?
Thanks for the help.

Couldn't it just be:
char keys[] = {
{'1', '2', '3', 'A'},
{'4', '5', '6', 'B'},
{'7', '8', '9', 'C'},
{'*', '0', '#', '0'}
};
Why isn't it?
It couldn't be because char keys[] specifies an array of type char. With the statement above, you are initializing each element to be an array of type char itself.
If you wanted to specify it as an array of type char, the correct way would be:
char keys[] = {
'1', '2', '3', 'A',
'4', '5', '6', 'B',
'7', '8', '9', 'C',
'*', '0', '#', '0'
};

Related

Handling Pagination with Scrapy

Helo all!
I am trying to scrap this website(https://codeblue.galencentre.org/malaysia/) using Scrapy.
I am having trouble getting the pagination and extracting the titles using the code below. Should I convert 'Content-Type: application/x-www-form-urlencoded' to 'application/json'? I am stuck half way.
Below is my attempt that yields a 'Error 400:HTTP status code is not handled or not allowed'.
import scrapy
from scrapy.http.request.form import FormRequest
class SihatSpider(scrapy.Spider):
name = 'sihat'
def start_requests(self):
query = 'query_params%5Bpost_types%5D=post&query_params%5Bi_attachment%5D=&query_params%5Btaxonomies%5D=category&query_params%5Bmulti_post_types%5D=&query_params%5Bmulti_taxonomies%5D=&query_params%5Bquery_types%5D=0&query_params%5Bi_taxonomies%5D=2%2C+51&query_params%5Be_taxonomies%5D=&query_params%5Bi_ids%5D=&query_params%5Bcq_operator%5D=0&query_params%5Be_ids%5D=&query_params%5Bquery_author%5D=&query_params%5Bquery_offset%5D=&query_params%5Bquery_include_children%5D=0&query_params%5Btoday_post%5D=0&query_params%5Bdatetime_meta%5D=&query_params%5Bwoo_query%5D=0&query_params%5Bpost_count%5D=-1&query_params%5Bposts_per_page%5D=19&query_params%5Bd_i_filter%5D=2&filter=&order=DESC&orderby=date&sub_opt_query%5Bmeta_key_query%5D=&sub_opt_query%5Bpaged%5D=1&sub_opt_query%5Bfirst_query%5D=off&sub_opt_query%5Btotal_pages%5D=181&sub_opt_query%5Bitems_last_page%5D=18&options%5Blayout_style%5D=3&options%5Bgrid_style%5D=0&options%5Blist_style%5D=0&options%5Bcarousel_t_style%5D=0&options%5Bcarousel_f_style%5D=0&options%5Bcreative_style%5D=0&options%5Btimeline_style%5D=0&options%5Bblock_content_style%5D=1&options%5Bgrid_masonry%5D=0&options%5Bshow_arrows%5D=1&options%5Barrows_outside%5D=0&options%5Bshow_dots%5D=1&options%5Binfinite%5D=1&options%5Bautoplay%5D=1&options%5Bautoplayspeed%5D=5000&options%5Bscrollperpage%5D=1&options%5Bspeed%5D=500&options%5Bcentermode%5D=0&options%5Bshow_elements%5D=0&options%5Bav_content%5D=0&options%5Bcc_mobile%5D=0&options%5Bcc_portrait_tablet%5D=0&options%5Bcc_landscape_tablet%5D=0&options%5Bcc_small_desktop%5D=0&options%5Bcc_medium_desktop%5D=0&options%5Bcc_large_desktop%5D=0&options%5Bcc_extra_large_desktop%5D=0&options%5Bimage_size%5D=thumbnail&options%5Bimage_size_s%5D=thumbnail&options%5Bs_image%5D=1&options%5Bs_image_link%5D=1&options%5Bs_image_link_target%5D=0&options%5Bs_icon_lightbox_video%5D=0&options%5Bvideo_url_meta%5D=0&options%5Bvideo_url_meta_key%5D=&options%5Bs_icon_lightbox_image%5D=0&options%5Bs_icon_link%5D=0&options%5Bs_icon_link_target%5D=0&options%5Bs_image_hover_effect%5D=1&options%5Bs_overlay_hover_effect%5D=&options%5Bs_overlay_settings%5D=0&options%5Bs_overlay_color%5D=&options%5Bs_image_post_format%5D=0&options%5Bs_image_post_format_pos%5D=ul-bottom-right&options%5Bs_title%5D=1&options%5Bs_title_limit%5D=0&options%5Bs_title_link%5D=1&options%5Bs_title_link_target%5D=0&options%5Bs_excerpt%5D=1&options%5Bs_excerpt_rbtn%5D=0&options%5Bs_excerpt_f%5D=get_the_excerpt&options%5Bs_excerpt_sc%5D=1&options%5Bs_excerpt_sh%5D=1&options%5Bs_excerpt_length%5D=0&options%5Bs_categories%5D=0&options%5Bs_s_categories%5D=0&options%5Bs_s_categories_parent%5D=0&options%5Bex_items_taxonomies%5D=&options%5Bs_c_categories%5D=0&options%5Bs_ct_categories%5D=&options%5Bs_cb_categories%5D=&options%5Bs_categories_target%5D=0&options%5Bs_metas_o%5D=1&options%5Bs_metas_o_author%5D=0&options%5Bs_metas_o_author_avatar%5D=0&options%5Bs_metas_o_time%5D=1&options%5Btime_format%5D=j+F%2C+Y&options%5Bs_metas_o_comment%5D=0&options%5Bs_metas_o_like%5D=0&options%5Bs_metas_o_share%5D=0&options%5Bcustom_meta_o%5D=&options%5Bs_metas_t%5D=0&options%5Bs_metas_t_author%5D=0&options%5Bs_metas_t_author_avatar%5D=0&options%5Bs_metas_t_time%5D=0&options%5Btime_format_t%5D=F+j%2C+Y&options%5Bs_metas_t_comment%5D=0&options%5Bs_metas_t_like%5D=1&options%5Bs_metas_t_share%5D=1&options%5Bcustom_meta_t%5D=&options%5Bs_metas_t_readmore%5D=1&options%5Bs_metas_t_readmore_link_target%5D=0&options%5Bshare_text%5D=&options%5Bread_more_text%5D=&options%5Bbefore_author_text%5D=&options%5Bpagination%5D=0&options%5Bloadmore_text%5D=&options%5Bprev_text%5D=&options%5Bnext_text%5D=&options%5Banimate%5D=default&options%5Blazyload%5D=0&options%5Blazyload_p%5D=&options%5Bgeodirectory_rating%5D=0&options%5Bquick_view%5D=1&options%5Bquick_view_mode%5D=0&options%5Bextra_class%5D=&options%5Brnd_id%5D=ul93310&options%5Bs_title_small%5D=1&options%5Bs_title_limit_small%5D=0&options%5Bs_title_link_small%5D=1&options%5Bs_title_link_target_small%5D=0&options%5Bs_categories_small%5D=0&options%5Bs_s_categories_small%5D=0&options%5Bs_s_categories_parent_small%5D=0&options%5Bex_items_taxonomies_small%5D=&options%5Bs_c_categories_small%5D=0&options%5Bs_ct_categories_small%5D=&options%5Bs_cb_categories_small%5D=&options%5Bs_categories_target_small%5D=0&options%5Bs_metas_o_small%5D=1&options%5Bs_metas_o_author_small%5D=0&options%5Bs_metas_o_author_avatar_small%5D=0&options%5Bs_metas_o_time_small%5D=1&options%5Btime_format_small%5D=j+F%2C+Y&options%5Bs_metas_o_comment_small%5D=0&options%5Bs_metas_o_like_small%5D=0&options%5Bs_metas_o_share_small%5D=0&options%5Bcustom_meta_o_small%5D=&options%5Bwoo_show_price%5D=0&options%5Bwoo_show_rating%5D=0&options%5Bwoo_show_cart%5D=0&options%5Bqv_s_title%5D=1&options%5Bqv_s_categories%5D=1&options%5Bqv_s_s_categories%5D=0&options%5Bqv_s_s_categories_parent%5D=0&options%5Bqv_ex_items_taxonomies%5D=&options%5Bqv_s_c_categories%5D=0&options%5Bqv_s_ct_categories%5D=&options%5Bqv_s_cb_categories%5D=&options%5Bqv_s_categories_target%5D=0&options%5Bqv_s_metas_o%5D=1&options%5Bqv_s_metas_o_author%5D=1&options%5Bqv_s_metas_o_author_avatar%5D=0&options%5Bqv_s_metas_o_time%5D=1&options%5Bqv_time_format%5D=F+j%2C+Y&options%5Bqv_s_metas_o_comment%5D=1&options%5Bqv_s_metas_o_like%5D=1&options%5Bqv_custom_meta_o%5D=&options%5Bqv_show_content%5D=1&options%5Bqv_content_stripsc%5D=0&options%5Bqv_show_share%5D=1&options%5Bqv_woo_show_rating%5D=1&options%5Bqv_s_featured_image%5D=1&options%5Bgoo_ads_client%5D=&options%5Bgoo_ads_id%5D=&options%5Bgoo_ads_offset%5D=1&options%5Bcss_class%5D=&random_id=ul93310&action=ultimatelayoutsajaxaction'
yield scrapy.FormRequest(
url = 'https://codeblue.galencentre.org/wp-admin/admin-ajax.php',
method='POST',
body=query,
headers={
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'x-requested-with': 'XMLHttpRequest'
},
callback=self.parse,
meta = {
'paged':1
}
)
def parse(self, response):
current_page = response.meta['paged']
for text in response.xpath("//div[#class='ultimate-layouts-content entry']"):
yield {
'date': text.xpath("./div[#class='ultimate-layouts-metas posted-on ']/div[#class='ultimate-layouts-metas-wrap']/div[#data-class='ul-time-metas']/span/text()").get(),
'title': text.xpath("./h3/a[#class='ultimate-layouts-title-link']/text()").get()
}
count = 181
current_page = 1
if current_page <= count:
current_page += 1
yield FormRequest(
url = 'https://codeblue.galencentre.org/wp-admin/admin-ajax.php',
method='POST',
body='query_params%5Bpost_types%5D=post&query_params%5Bi_attachment%5D=&query_params%5Btaxonomies%5D=category&query_params%5Bmulti_post_types%5D=&query_params%5Bmulti_taxonomies%5D=&query_params%5Bquery_types%5D=0&query_params%5Bi_taxonomies%5D=2%2C+51&query_params%5Be_taxonomies%5D=&query_params%5Bi_ids%5D=&query_params%5Bcq_operator%5D=0&query_params%5Be_ids%5D=&query_params%5Bquery_author%5D=&query_params%5Bquery_offset%5D=&query_params%5Bquery_include_children%5D=0&query_params%5Btoday_post%5D=0&query_params%5Bdatetime_meta%5D=&query_params%5Bwoo_query%5D=0&query_params%5Bpost_count%5D=-1&query_params%5Bposts_per_page%5D=19&query_params%5Bd_i_filter%5D=2&filter=&order=DESC&orderby=date&sub_opt_query%5Bmeta_key_query%5D=&sub_opt_query%5Bpaged%5D={current_page}&sub_opt_query%5Bfirst_query%5D=off&sub_opt_query%5Btotal_pages%5D=181&sub_opt_query%5Bitems_last_page%5D=18&options%5Blayout_style%5D=3&options%5Bgrid_style%5D=0&options%5Blist_style%5D=0&options%5Bcarousel_t_style%5D=0&options%5Bcarousel_f_style%5D=0&options%5Bcreative_style%5D=0&options%5Btimeline_style%5D=0&options%5Bblock_content_style%5D=1&options%5Bgrid_masonry%5D=0&options%5Bshow_arrows%5D=1&options%5Barrows_outside%5D=0&options%5Bshow_dots%5D=1&options%5Binfinite%5D=1&options%5Bautoplay%5D=1&options%5Bautoplayspeed%5D=5000&options%5Bscrollperpage%5D=1&options%5Bspeed%5D=500&options%5Bcentermode%5D=0&options%5Bshow_elements%5D=0&options%5Bav_content%5D=0&options%5Bcc_mobile%5D=0&options%5Bcc_portrait_tablet%5D=0&options%5Bcc_landscape_tablet%5D=0&options%5Bcc_small_desktop%5D=0&options%5Bcc_medium_desktop%5D=0&options%5Bcc_large_desktop%5D=0&options%5Bcc_extra_large_desktop%5D=0&options%5Bimage_size%5D=thumbnail&options%5Bimage_size_s%5D=thumbnail&options%5Bs_image%5D=1&options%5Bs_image_link%5D=1&options%5Bs_image_link_target%5D=0&options%5Bs_icon_lightbox_video%5D=0&options%5Bvideo_url_meta%5D=0&options%5Bvideo_url_meta_key%5D=&options%5Bs_icon_lightbox_image%5D=0&options%5Bs_icon_link%5D=0&options%5Bs_icon_link_target%5D=0&options%5Bs_image_hover_effect%5D=1&options%5Bs_overlay_hover_effect%5D=&options%5Bs_overlay_settings%5D=0&options%5Bs_overlay_color%5D=&options%5Bs_image_post_format%5D=0&options%5Bs_image_post_format_pos%5D=ul-bottom-right&options%5Bs_title%5D=1&options%5Bs_title_limit%5D=0&options%5Bs_title_link%5D=1&options%5Bs_title_link_target%5D=0&options%5Bs_excerpt%5D=1&options%5Bs_excerpt_rbtn%5D=0&options%5Bs_excerpt_f%5D=get_the_excerpt&options%5Bs_excerpt_sc%5D=1&options%5Bs_excerpt_sh%5D=1&options%5Bs_excerpt_length%5D=0&options%5Bs_categories%5D=0&options%5Bs_s_categories%5D=0&options%5Bs_s_categories_parent%5D=0&options%5Bex_items_taxonomies%5D=&options%5Bs_c_categories%5D=0&options%5Bs_ct_categories%5D=&options%5Bs_cb_categories%5D=&options%5Bs_categories_target%5D=0&options%5Bs_metas_o%5D=1&options%5Bs_metas_o_author%5D=0&options%5Bs_metas_o_author_avatar%5D=0&options%5Bs_metas_o_time%5D=1&options%5Btime_format%5D=j+F%2C+Y&options%5Bs_metas_o_comment%5D=0&options%5Bs_metas_o_like%5D=0&options%5Bs_metas_o_share%5D=0&options%5Bcustom_meta_o%5D=&options%5Bs_metas_t%5D=0&options%5Bs_metas_t_author%5D=0&options%5Bs_metas_t_author_avatar%5D=0&options%5Bs_metas_t_time%5D=0&options%5Btime_format_t%5D=F+j%2C+Y&options%5Bs_metas_t_comment%5D=0&options%5Bs_metas_t_like%5D=1&options%5Bs_metas_t_share%5D=1&options%5Bcustom_meta_t%5D=&options%5Bs_metas_t_readmore%5D=1&options%5Bs_metas_t_readmore_link_target%5D=0&options%5Bshare_text%5D=&options%5Bread_more_text%5D=&options%5Bbefore_author_text%5D=&options%5Bpagination%5D=0&options%5Bloadmore_text%5D=&options%5Bprev_text%5D=&options%5Bnext_text%5D=&options%5Banimate%5D=default&options%5Blazyload%5D=0&options%5Blazyload_p%5D=&options%5Bgeodirectory_rating%5D=0&options%5Bquick_view%5D=1&options%5Bquick_view_mode%5D=0&options%5Bextra_class%5D=&options%5Brnd_id%5D=ul93310&options%5Bs_title_small%5D=1&options%5Bs_title_limit_small%5D=0&options%5Bs_title_link_small%5D=1&options%5Bs_title_link_target_small%5D=0&options%5Bs_categories_small%5D=0&options%5Bs_s_categories_small%5D=0&options%5Bs_s_categories_parent_small%5D=0&options%5Bex_items_taxonomies_small%5D=&options%5Bs_c_categories_small%5D=0&options%5Bs_ct_categories_small%5D=&options%5Bs_cb_categories_small%5D=&options%5Bs_categories_target_small%5D=0&options%5Bs_metas_o_small%5D=1&options%5Bs_metas_o_author_small%5D=0&options%5Bs_metas_o_author_avatar_small%5D=0&options%5Bs_metas_o_time_small%5D=1&options%5Btime_format_small%5D=j+F%2C+Y&options%5Bs_metas_o_comment_small%5D=0&options%5Bs_metas_o_like_small%5D=0&options%5Bs_metas_o_share_small%5D=0&options%5Bcustom_meta_o_small%5D=&options%5Bwoo_show_price%5D=0&options%5Bwoo_show_rating%5D=0&options%5Bwoo_show_cart%5D=0&options%5Bqv_s_title%5D=1&options%5Bqv_s_categories%5D=1&options%5Bqv_s_s_categories%5D=0&options%5Bqv_s_s_categories_parent%5D=0&options%5Bqv_ex_items_taxonomies%5D=&options%5Bqv_s_c_categories%5D=0&options%5Bqv_s_ct_categories%5D=&options%5Bqv_s_cb_categories%5D=&options%5Bqv_s_categories_target%5D=0&options%5Bqv_s_metas_o%5D=1&options%5Bqv_s_metas_o_author%5D=1&options%5Bqv_s_metas_o_author_avatar%5D=0&options%5Bqv_s_metas_o_time%5D=1&options%5Bqv_time_format%5D=F+j%2C+Y&options%5Bqv_s_metas_o_comment%5D=1&options%5Bqv_s_metas_o_like%5D=1&options%5Bqv_custom_meta_o%5D=&options%5Bqv_show_content%5D=1&options%5Bqv_content_stripsc%5D=0&options%5Bqv_show_share%5D=1&options%5Bqv_woo_show_rating%5D=1&options%5Bqv_s_featured_image%5D=1&options%5Bgoo_ads_client%5D=&options%5Bgoo_ads_id%5D=&options%5Bgoo_ads_offset%5D=1&options%5Bcss_class%5D=&random_id=ul93310&action=ultimatelayoutsajaxaction',
headers={
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'x-requested-with': 'XMLHttpRequest'
},
callback=self.parse,
meta = {
'paged':current_page
}
)
It looks like you're not actually getting to the next page at all.
All you have really done is set up an integer to count from 1 to 183, without a direct execution to changing the pages.
One way of doing what you want is to checkout the payload.
This line here:
sub_opt_query[paged]: 2
gives the page number, so you just include the entire payload removing those without any values and assorting everything as strings.
With this in mind, we get the following scraper that get's all the pages respectively:
import scrapy
from scrapy.http.request.form import FormRequest
from scrapy.crawler import CrawlerProcess
class SihatSpider(scrapy.Spider):
name = 'sihat'
def start_requests(self):
query = 'query_params%5Bpost_types%5D=post&query_params%5Bi_attachment%5D=&query_params%5Btaxonomies%5D=category&query_params%5Bmulti_post_types%5D=&query_params%5Bmulti_taxonomies%5D=&query_params%5Bquery_types%5D=0&query_params%5Bi_taxonomies%5D=2%2C+51&query_params%5Be_taxonomies%5D=&query_params%5Bi_ids%5D=&query_params%5Bcq_operator%5D=0&query_params%5Be_ids%5D=&query_params%5Bquery_author%5D=&query_params%5Bquery_offset%5D=&query_params%5Bquery_include_children%5D=0&query_params%5Btoday_post%5D=0&query_params%5Bdatetime_meta%5D=&query_params%5Bwoo_query%5D=0&query_params%5Bpost_count%5D=-1&query_params%5Bposts_per_page%5D=19&query_params%5Bd_i_filter%5D=2&filter=&order=DESC&orderby=date&sub_opt_query%5Bmeta_key_query%5D=&sub_opt_query%5Bpaged%5D=1&sub_opt_query%5Bfirst_query%5D=off&sub_opt_query%5Btotal_pages%5D=181&sub_opt_query%5Bitems_last_page%5D=18&options%5Blayout_style%5D=3&options%5Bgrid_style%5D=0&options%5Blist_style%5D=0&options%5Bcarousel_t_style%5D=0&options%5Bcarousel_f_style%5D=0&options%5Bcreative_style%5D=0&options%5Btimeline_style%5D=0&options%5Bblock_content_style%5D=1&options%5Bgrid_masonry%5D=0&options%5Bshow_arrows%5D=1&options%5Barrows_outside%5D=0&options%5Bshow_dots%5D=1&options%5Binfinite%5D=1&options%5Bautoplay%5D=1&options%5Bautoplayspeed%5D=5000&options%5Bscrollperpage%5D=1&options%5Bspeed%5D=500&options%5Bcentermode%5D=0&options%5Bshow_elements%5D=0&options%5Bav_content%5D=0&options%5Bcc_mobile%5D=0&options%5Bcc_portrait_tablet%5D=0&options%5Bcc_landscape_tablet%5D=0&options%5Bcc_small_desktop%5D=0&options%5Bcc_medium_desktop%5D=0&options%5Bcc_large_desktop%5D=0&options%5Bcc_extra_large_desktop%5D=0&options%5Bimage_size%5D=thumbnail&options%5Bimage_size_s%5D=thumbnail&options%5Bs_image%5D=1&options%5Bs_image_link%5D=1&options%5Bs_image_link_target%5D=0&options%5Bs_icon_lightbox_video%5D=0&options%5Bvideo_url_meta%5D=0&options%5Bvideo_url_meta_key%5D=&options%5Bs_icon_lightbox_image%5D=0&options%5Bs_icon_link%5D=0&options%5Bs_icon_link_target%5D=0&options%5Bs_image_hover_effect%5D=1&options%5Bs_overlay_hover_effect%5D=&options%5Bs_overlay_settings%5D=0&options%5Bs_overlay_color%5D=&options%5Bs_image_post_format%5D=0&options%5Bs_image_post_format_pos%5D=ul-bottom-right&options%5Bs_title%5D=1&options%5Bs_title_limit%5D=0&options%5Bs_title_link%5D=1&options%5Bs_title_link_target%5D=0&options%5Bs_excerpt%5D=1&options%5Bs_excerpt_rbtn%5D=0&options%5Bs_excerpt_f%5D=get_the_excerpt&options%5Bs_excerpt_sc%5D=1&options%5Bs_excerpt_sh%5D=1&options%5Bs_excerpt_length%5D=0&options%5Bs_categories%5D=0&options%5Bs_s_categories%5D=0&options%5Bs_s_categories_parent%5D=0&options%5Bex_items_taxonomies%5D=&options%5Bs_c_categories%5D=0&options%5Bs_ct_categories%5D=&options%5Bs_cb_categories%5D=&options%5Bs_categories_target%5D=0&options%5Bs_metas_o%5D=1&options%5Bs_metas_o_author%5D=0&options%5Bs_metas_o_author_avatar%5D=0&options%5Bs_metas_o_time%5D=1&options%5Btime_format%5D=j+F%2C+Y&options%5Bs_metas_o_comment%5D=0&options%5Bs_metas_o_like%5D=0&options%5Bs_metas_o_share%5D=0&options%5Bcustom_meta_o%5D=&options%5Bs_metas_t%5D=0&options%5Bs_metas_t_author%5D=0&options%5Bs_metas_t_author_avatar%5D=0&options%5Bs_metas_t_time%5D=0&options%5Btime_format_t%5D=F+j%2C+Y&options%5Bs_metas_t_comment%5D=0&options%5Bs_metas_t_like%5D=1&options%5Bs_metas_t_share%5D=1&options%5Bcustom_meta_t%5D=&options%5Bs_metas_t_readmore%5D=1&options%5Bs_metas_t_readmore_link_target%5D=0&options%5Bshare_text%5D=&options%5Bread_more_text%5D=&options%5Bbefore_author_text%5D=&options%5Bpagination%5D=0&options%5Bloadmore_text%5D=&options%5Bprev_text%5D=&options%5Bnext_text%5D=&options%5Banimate%5D=default&options%5Blazyload%5D=0&options%5Blazyload_p%5D=&options%5Bgeodirectory_rating%5D=0&options%5Bquick_view%5D=1&options%5Bquick_view_mode%5D=0&options%5Bextra_class%5D=&options%5Brnd_id%5D=ul93310&options%5Bs_title_small%5D=1&options%5Bs_title_limit_small%5D=0&options%5Bs_title_link_small%5D=1&options%5Bs_title_link_target_small%5D=0&options%5Bs_categories_small%5D=0&options%5Bs_s_categories_small%5D=0&options%5Bs_s_categories_parent_small%5D=0&options%5Bex_items_taxonomies_small%5D=&options%5Bs_c_categories_small%5D=0&options%5Bs_ct_categories_small%5D=&options%5Bs_cb_categories_small%5D=&options%5Bs_categories_target_small%5D=0&options%5Bs_metas_o_small%5D=1&options%5Bs_metas_o_author_small%5D=0&options%5Bs_metas_o_author_avatar_small%5D=0&options%5Bs_metas_o_time_small%5D=1&options%5Btime_format_small%5D=j+F%2C+Y&options%5Bs_metas_o_comment_small%5D=0&options%5Bs_metas_o_like_small%5D=0&options%5Bs_metas_o_share_small%5D=0&options%5Bcustom_meta_o_small%5D=&options%5Bwoo_show_price%5D=0&options%5Bwoo_show_rating%5D=0&options%5Bwoo_show_cart%5D=0&options%5Bqv_s_title%5D=1&options%5Bqv_s_categories%5D=1&options%5Bqv_s_s_categories%5D=0&options%5Bqv_s_s_categories_parent%5D=0&options%5Bqv_ex_items_taxonomies%5D=&options%5Bqv_s_c_categories%5D=0&options%5Bqv_s_ct_categories%5D=&options%5Bqv_s_cb_categories%5D=&options%5Bqv_s_categories_target%5D=0&options%5Bqv_s_metas_o%5D=1&options%5Bqv_s_metas_o_author%5D=1&options%5Bqv_s_metas_o_author_avatar%5D=0&options%5Bqv_s_metas_o_time%5D=1&options%5Bqv_time_format%5D=F+j%2C+Y&options%5Bqv_s_metas_o_comment%5D=1&options%5Bqv_s_metas_o_like%5D=1&options%5Bqv_custom_meta_o%5D=&options%5Bqv_show_content%5D=1&options%5Bqv_content_stripsc%5D=0&options%5Bqv_show_share%5D=1&options%5Bqv_woo_show_rating%5D=1&options%5Bqv_s_featured_image%5D=1&options%5Bgoo_ads_client%5D=&options%5Bgoo_ads_id%5D=&options%5Bgoo_ads_offset%5D=1&options%5Bcss_class%5D=&random_id=ul93310&action=ultimatelayoutsajaxaction'
for i in range(1, 183, 1):
yield scrapy.FormRequest(
url = 'https://codeblue.galencentre.org/wp-admin/admin-ajax.php',
method='POST',
body=query,
headers={
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'x-requested-with': 'XMLHttpRequest'
},
formdata = {'query_params[post_types]': 'post',
'query_params[taxonomies]': 'category',
'query_params[query_types]': '0',
'query_params[cq_operator]': '0',
'query_params[query_include_children]': '0',
'query_params[today_post]': '0',
'query_params[woo_query]': '0',
'query_params[post_count]': '-1',
'query_params[posts_per_page]': '19',
'query_params[d_i_filter]': '2',
'order': 'DESC',
'orderby': 'date',
'sub_opt_query[paged]': str(i),
'sub_opt_query[first_query]': 'off',
'sub_opt_query[total_pages]': '183',
'sub_opt_query[items_last_page]': '2',
'options[layout_style]': '3',
'options[grid_style]': '0',
'options[list_style]': '0',
'options[carousel_t_style]': '0',
'options[carousel_f_style]': '0',
'options[creative_style]': '0',
'options[timeline_style]': '0',
'options[block_content_style]': '1',
'options[grid_masonry]': '0',
'options[show_arrows]': '1',
'options[arrows_outside]': '0',
'options[show_dots]': '1',
'options[infinite]': '1',
'options[autoplay]': '1',
'options[autoplayspeed]': '5000',
'options[scrollperpage]': '1',
'options[speed]': '500',
'options[centermode]': '0',
'options[show_elements]': '0',
'options[av_content]': '0',
'options[cc_mobile]': '0',
'options[cc_portrait_tablet]': '0',
'options[cc_landscape_tablet]': '0',
'options[cc_small_desktop]': '0',
'options[cc_medium_desktop]': '0',
'options[cc_large_desktop]': '0',
'options[cc_extra_large_desktop]': '0',
'options[image_size]': 'thumbnail',
'options[image_size_s]': 'thumbnail',
'options[s_image]': '1',
'options[s_image_link]': '1',
'options[s_image_link_target]': '0',
'options[s_icon_lightbox_video]': '0',
'options[video_url_meta]': '0',
'options[s_icon_lightbox_image]': '0',
'options[s_icon_link]': '0',
'options[s_icon_link_target]': '0',
'options[s_image_hover_effect]': '1',
'options[s_overlay_settings]': '0',
'options[s_image_post_format]': '0',
'options[s_image_post_format_pos]': 'ul-bottom-right',
'options[s_title]': '1',
'options[s_title_limit]': '0',
'options[s_title_link]': '1',
'options[s_title_link_target]': '0',
'options[s_excerpt]': '1',
'options[s_excerpt_rbtn]': '0',
'options[s_excerpt_f]': 'get_the_excerpt',
'options[s_excerpt_sc]': '1',
'options[s_excerpt_sh]': '1',
'options[s_excerpt_length]': '0',
'options[s_categories]': '0',
'options[s_s_categories]': '0',
'options[s_s_categories_parent]': '0',
'options[s_c_categories]': '0',
'options[s_categories_target]': '0',
'options[s_metas_o]': '1',
'options[s_metas_o_author]': '0',
'options[s_metas_o_author_avatar]': '0',
'options[s_metas_o_time]': '1',
'options[time_format]': 'j F, Y',
'options[s_metas_o_comment]': '0',
'options[s_metas_o_like]': '0',
'options[s_metas_o_share]': '0',
'options[s_metas_t]': '0',
'options[s_metas_t_author]': '0',
'options[s_metas_t_author_avatar]': '0',
'options[s_metas_t_time]': '0',
'options[time_format_t]': 'F j, Y',
'options[s_metas_t_comment]': '0',
'options[s_metas_t_like]': '1',
'options[s_metas_t_share]': '1',
'options[s_metas_t_readmore]': '1',
'options[s_metas_t_readmore_link_target]': '0',
'options[pagination]': '0',
'options[animate]': 'default',
'options[lazyload]': '0',
'options[geodirectory_rating]': '0',
'options[quick_view]': '1',
'options[quick_view_mode]': '0',
'options[rnd_id]': 'ul32943',
'options[s_title_small]': '1',
'options[s_title_limit_small]': '0',
'options[s_title_link_small]': '1',
'options[s_title_link_target_small]': '0',
'options[s_categories_small]': '0',
'options[s_s_categories_small]': '0',
'options[s_s_categories_parent_small]': '0',
'options[s_c_categories_small]': '0',
'options[s_categories_target_small]': '0',
'options[s_metas_o_small]': '1',
'options[s_metas_o_author_small]': '0',
'options[s_metas_o_author_avatar_small]': '0',
'options[s_metas_o_time_small]': '1',
'options[time_format_small]': 'j F, Y',
'options[s_metas_o_comment_small]': '0',
'options[s_metas_o_like_small]': '0',
'options[s_metas_o_share_small]': '0',
'options[woo_show_price]': '0',
'options[woo_show_rating]': '0',
'options[woo_show_cart]': '0',
'options[qv_s_title]': '1',
'options[qv_s_categories]': '1',
'options[qv_s_s_categories]': '0',
'options[qv_s_s_categories_parent]': '0',
'options[qv_s_c_categories]': '0',
'options[qv_s_categories_target]': '0',
'options[qv_s_metas_o]': '1',
'options[qv_s_metas_o_author]': '1',
'options[qv_s_metas_o_author_avatar]': '0',
'options[qv_s_metas_o_time]': '1',
'options[qv_time_format]': 'F j, Y',
'options[qv_s_metas_o_comment]': '1',
'options[qv_s_metas_o_like]': '1',
'options[qv_show_content]': '1',
'options[qv_content_stripsc]': '0',
'options[qv_show_share]': '1',
'options[qv_woo_show_rating]': '1',
'options[qv_s_featured_image]': '1',
'options[goo_ads_offset]': '1',
'random_id': 'ul32943',
'action': 'ultimatelayoutsajaxaction'},
callback=self.parse,
)
def parse(self, response):
for text in response.xpath("//div[#class='ultimate-layouts-content entry']"):
yield {
'date': text.xpath("./div[#class='ultimate-layouts-metas posted-on ']/div[#class='ultimate-layouts-metas-wrap']/div[#data-class='ul-time-metas']/span/text()").get(),
'title': text.xpath("./h3/a[#class='ultimate-layouts-title-link']/text()").get()
}
process = CrawlerProcess(
settings={
'FEED_URI': 'Media_content.jl',
'FEED_FORMAT': 'jsonlines'
}
)
process.crawl(SihatSpider)
process.start()

Cosmos DB - Query for newest document of select partitions?

Consider a CosmosDB container with the following document model:
{
id: <string>,
userId: <string>, // partition key
data: <string>
}
I have a need to provide a query with N user ids and get the newest document for each one.
So for example, if I have this data in the container:
{ id: '1', userId: 'user1', data: 'a', _ts: 1 },
{ id: '2', userId: 'user1', data: 'b', _ts: 2 },
{ id: '3', userId: 'user2', data: 'c', _ts: 10 },
{ id: '4', userId: 'user2', data: 'd', _ts: 5 },
{ id: '5', userId: 'user3', data: 'e', _ts: 3 },
{ id: '6', userId: 'user3', data: 'f', _ts: 4 },
{ id: '7', userId: 'user4', data: 'g', _ts: 100 },
{ id: '8', userId: 'user4', data: 'h', _ts: 99 },
{ id: '9', userId: 'user5', data: 'i', _ts: 1 },
{ id: '10', userId: 'user5', data: 'j', _ts: 2 },
I want to do something like this:
-- This doesn't work
SELECT c.userId, (SELECT TOP 1 d.id, d.data WHERE d.userId = c.userId FROM d ORDER BY d._ts DESC) AS newest
WHERE c.userId IN ['user1', 'user2', 'user4', 'user5']
To get this result:
{ userId: 'user1', newest: { id: '2', data: 'b' } },
{ userId: 'user2', newest: { id: '3', data: 'c' } },
{ userId: 'user4', newest: { id: '7', data: 'g' } },
{ userId: 'user5', newest: { id: '10', data: 'j' } },
From what I can tell, JOIN in CosmosDB cannot be used to filter correlated documents. Is there still a way to accomplish this? I am open to using a stored procedure, but from what I can tell execution of a stored procedure can only occur on a specific partition given it's key. In my case, the primary grouping is the partition key.
I have considered a fan-out request approach, but I might be querying for 50 to 100 user ids at a time in the query. In that case it might be faster to just get all documents in each partition and when iterating only keep the newest -- but that's a large paged response to sift through.
My final thought is I could use ASB/EventGrid/Function and another dependent CosmosDB container to always clone the most recent updated document every time a document is updated, but it seems like overkill. Surely there is a way to construct a query to do what I want?
Thanks
I have an idea like
select c._ts from c where ARRAY_CONTAINS((select value max(c._ts) from c group by c.userId), c._ts)
But it can't get the result because select value max(c._ts) from c group by c.userId doesn't be recognized as an array, and if I use Array(select value max(c._ts) from c group by c.userId) instead, it returns all items.
So how about execute sql twice?
Get the timestamp array first :select value max(c._ts) from c where c.userId in ('user1','user2') group by c.userId
,
and then copy the result as the input to use array_contains function:
select c._ts,c.data from c where ARRAY_CONTAINS([1623306298,1623306259,1623306217], c._ts)
One way of doing this would be to use the following approach.
SELECT t.userid,
SUBSTRING(t.concat, 28,8000) AS data
FROM
(
SELECT c.userid,
MAX(CONCAT(TimestampToDateTime(c._ts*1000),c.data)) AS concat
FROM c
WHERE c.userid IN ('user1', 'user2')
GROUP BY c.userid
) AS t
which returns a result like
[
{
"userid": "user1",
"data": "b"
},
{
"userid": "user2",
"data": "d"
}
]
The derived table t returns results like the following...
[
{
"userid": "user2",
"concat": "2021-06-11T17:42:03.0000000Zd"
},
{
"userid": "user1",
"concat": "2021-06-11T17:41:41.0000000Zb"
}
]
The document with the highest _ts per user will have the lexicographically highest datetime prefix in the concatenated string and the ancillary data that is appended on behind it is extracted with SUBSTRING.
It should be able to use the index for the WHERE clause - but then will need to look at all documents for the given userids (so if there are many documents per user doing separate TOP 1 queries for each one would likely be much better)

Create method like System.Web.Helpers.Crypto.HashPassword (ASP.NET) in nodejs?

How can I make password hash using RFC 2898 like https://learn.microsoft.com/en-us/previous-versions/aspnet/web-frameworks/gg538287(v=vs.111) in nodejs?
My nodejs app are using a table of SQL server which have password field hashed by Crypto.HashPassword of ASP.NET, so I need create same function in nodejs to compare it.
const crypto = require('crypto');
const hexChar = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F'];
const verifyHashedPassword = (password, hashedPwd) => {
let saltString = '';
let storedSubKeyString = '';
const hashedPasswordBytes = new Buffer(hashedPwd, 'base64');
for (var i = 1; i < hashedPasswordBytes.length; i++) {
if (i > 0 && i <= 16) {
saltString += hexChar[(hashedPasswordBytes[i] >> 4) & 0x0f] + hexChar[hashedPasswordBytes[i] & 0x0f];
}
if (i > 0 && i > 16) {
storedSubKeyString += hexChar[(hashedPasswordBytes[i] >> 4) & 0x0f] + hexChar[hashedPasswordBytes[i] & 0x0f];
}
}
const nodeCrypto = crypto.pbkdf2Sync(new Buffer(password), new Buffer(saltString, 'hex'), 1000, 256, 'sha1');
const derivedKeyOctets = nodeCrypto.toString('hex').toUpperCase();
return derivedKeyOctets.indexOf(storedSubKeyString) === 0;
};
I used that for compare plain password with hashed password. It's working well!

What do I need to use in this? Another cursor, a record or something else?

The code below is the answer from my question yesterday that has the ff results:
First run
Updated: 0
Inserted: 4
2nd run
Updated: 4
Inserted: 0
After deleting cid '1' and '3':
Updated: 2
Inserted: 2
DECLARE
ins NUMBER := 0;
upd NUMBER := 0;
CURSOR c1 IS
SELECT cid
FROM tbl_cust
WHERE cid
IN ('1','2','3','4');
BEGIN
FOR rec IN c1 LOOP
begin
INSERT INTO tbl2 (id_tbl2, name_tbl2)
VALUES(rec.cid, DECODE(rec.cid, '1', 'A',
'2', 'B',
'3', 'C',
'4', 'D'));
ins := ins + 1;
EXCEPTION WHEN DUP_VAL_ON_INDEX THEN
UPDATE tbl2 set name_tbl2 = DECODE(rec.cid, '1', 'A',
'2', 'B',
'3', 'C',
'4', 'D'));
WHERE cust_cust_code = rec.cid;
upd := upd + 1;
continue;
end;
END LOOP;
dbms_output.put_line('Updated: ' || upd);
dbms_output.put_line('Inserted: ' || ins);
END;
What I wanted now is to revise this code without using nested block. Maybe something like this:
DECLARE
ins NUMBER := 0;
upd NUMBER := 0;
CURSOR c1 IS
SELECT cid
FROM tbl_cust
WHERE cid
IN ('1','2','3','4');
--maybe declare something as 'holder of values' which are ducplicates and cannot be inserted then will be used on a condition or loop to update.
BEGIN
FOR rec IN c1 LOOP
INSERT INTO tbl2 (id_tbl2, name_tbl2)
VALUES(rec.cid, DECODE(rec.cid, '1', 'A',
'2', 'B',
'3', 'C',
'4', 'D'));
ins := ins + 1;
--maybe put some condition here to pass the values that are not insert to the 'holder of values'
END LOOP;
dbms_output.put_line('Updated: ' || upd);
dbms_output.put_line('Inserted: ' || ins);
EXCEPTION
WHEN DUP_VAL_ON_INDEX THEN
--maybe put a loop here to update values in the 'holder of values'
UPDATE tbl2 set name_tbl2 = DECODE('holder of values'.VALUE, '1', 'A',
'2', 'B',
'3', 'C',
'4', 'D')
WHERE cust_cust_code = 'holder of values'.VALUE;
upd := upd + 1;
dbms_output.put_line('Updated: ' || upd);
dbms_output.put_line('Inserted: ' || ins);
END;
Actually the best way to do this is merge as was suggested by so many others.
(and first to mention this was #APC)
However if you insist on your logic then you should use some sort of collection. (take a look here Working with Collections).
Nevertheless, you cannot put exception handling outside the loop it still should be inside.
Here how this could look like:
DECLARE
ins NUMBER := 0;
upd NUMBER := 0;
CURSOR c1 IS
SELECT cid
FROM tbl_cust
WHERE cid
IN ('1','2','3','4');
--declare something as 'holder of values'
TYPE list_of_ids IS TABLE OF number;
duplicates list_of_ids:= list_of_ids ();
BEGIN
FOR rec IN c1 LOOP
begin
INSERT INTO tbl2 (id_tbl2, name_tbl2)
VALUES(rec.cid, DECODE(rec.cid, '1', 'A',
'2', 'B',
'3', 'C',
'4', 'D'));
ins := ins + sql%rowcount;
EXCEPTION WHEN DUP_VAL_ON_INDEX THEN
duplicates.EXTEND;
duplicates(duplicates.LAST):=rec.cid;
end;
--condition here to pass the values that are not insert to the 'holder of values'
END LOOP;
dbms_output.put_line('Updated: ' || upd);
dbms_output.put_line('Inserted: ' || ins);
--a loop here to update values in the 'holder of values'
FOR l_row IN 1 .. duplicates.COUNT
LOOP
UPDATE tbl2 set name_tbl2 = DECODE(duplicates (l_row), '1', 'A',
'2', 'B',
'3', 'C',
'4', 'D')
WHERE cust_cust_code := duplicates (l_row);
upd := upd + sql%rowcount;
DBMS_OUTPUT.put_line (duplicates (l_row));
END LOOP;
dbms_output.put_line('Updated: ' || upd);
dbms_output.put_line('Inserted: ' || ins);
END;

Remove default optgroup label '0' in array select using Symfony2

I have a form that shows a drop-down menu to select a single value from an array defined in params.yml.
The dropdown displays '0' as an optgroup label above the list of values to be selected.
How can I get rid of this default label ?
You can replace the default OptGroup with something like the "Standard Hours" and "Other Hours" as defined below.
$category_choices = array(
'Standard Hours' => array(
2 => '2',
4 => '4',
6 => '6',
8 => '8'
),
'Other Hours' => array(
1 => '1',
3 => '3',
5 => '5',
7 => '7',
9 => '9',
10 => '10'
)
);
$builder
->add('hours', 'choice', array(
'choices' => $category_choices
));
Alternatively, if you don't want OptGroup labels at all, and only selectable entries, you can just skip out on the nested arrays altogether.
$builder
->add('hours', 'choice', array(
'choices' => array(
1 => '1',
2 => '2',
3 => '3',
4 => '4'
)
));
In your case, you want something like:
$builder
->add('hours', 'choice', array(
'choices' => arrayFromParamsYML
));

Resources