F# AWS lambda - Parts of asynchronous function are never executed - asynchronous
I have an asynchronous F# function in an AWS Lambda. However, when testing it, parts of the function are never called. The function is always terminated after 10 seconds, even though the aws-lambda-tools-defaults.json file specifies the timeout to 30 seconds.
Here is the code:
namespace MyProject
open FSharp.Data
type SecretsJson = JsonProvider<"./Resources/Secrets.json", RootName="Secret">
module ApiClient =
let testGet (secrets: SecretsJson.Secret) =
async {
printfn "%s" "3. This is sometimes printed out and sometimes not."
let! test = Http.AsyncRequestString("https://postman-echo.com/get?foo1=bar1&foo2=bar2")
printfn "%s" "4. This is never printed."
return ""
}
namespace MyProject
open Amazon.Lambda.Core
open Amazon
open Amazon.S3
open Amazon.S3.Util
open System.IO
open Amazon.S3.Model
open Amazon.SecretsManager.Extensions.Caching
[<assembly: LambdaSerializer(typeof<Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer>)>]
()
type Function() =
member __.FunctionHandler (input: S3EventNotification) (_: ILambdaContext) : System.Threading.Tasks.Task<unit> =
printfn "%s" "1. This gets printed always."
async {
use client = new AmazonS3Client(RegionEndpoint.EUWest1)
use secretsCache = new SecretsManagerCache()
let! secretsString = secretsCache.GetSecretString "secretsKey" |> Async.AwaitTask
printfn "%s" "2. Also this is always printed."
let secrets = SecretsJson.Parse(secretsString)
let! response = ApiClient.testGet secrets
printfn "%s" "5. This is never printed."
} |> Async.StartAsTask
I don't understand this behaviour. I thought that the |> Async.StartAsTask expression will ensure that the whole async block in the Function.FunctionHandler gets executed as a standard C# Task<T>. Or do I have to convert each of my Async<'a> functions into a Task<T>? Or is there a different error in my code that I don't see?
The async code shouldn't be the source of the problem. I tried the following:
open System
open Amazon.Lambda.Core
open FSharp.Data
let loadStuff (context: ILambdaContext) (i:int) =
async {
let! test = Http.AsyncRequestString("https://www.google.com")
sprintf "i:%i Length: %i" i test.Length
|> context.Logger.LogLine
}
and
type Functions() =
member __.Get (request: APIGatewayProxyRequest) (context: ILambdaContext) =
async {
do! [1..10]
|>Seq.map (Loader.loadStuff context)
|>Async.Parallel
|>Async.Ignore
do! Async.Sleep(10000)
do! [11..15]
|>Seq.map (Loader.loadStuff context)
|>Async.Parallel
|>Async.Ignore
}
|> Async.StartAsTask
This takes 12 seconds and outputs:
i:6 Length: 47974
i:3 Length: 47201
i:4 Length: 47200
i:8 Length: 47255
i:5 Length: 47183
i:1 Length: 47145
i:7 Length: 47203
i:9 Length: 47177
i:10 Length: 47202
i:2 Length: 47198
i:14 Length: 47201
i:12 Length: 47155
i:11 Length: 47250
i:13 Length: 47162
i:15 Length: 47130
Consider adding try catch blocks to your code and also using context.Logger.Logline instead of printfn. Also if you have very long running pieces of work e.g. >30secs. Consider breaking them down into more functions and possibly coordinating them with something like serverless workflows.
Related
Assembly.LoadFrom fails but only sporadically
I want to write a program that watches a .dll for changes. When a change happens, it should load the assembly and invoke the foo function inside. I have some code that should implement this, but it behaves strangely. Sometimes it works. Sometimes the assembly it loads will be an old version. Sometimes it will throw a BadImageFormatException exception. Here is my program code (it is F# but I think this is a general .NET Core question): module HotReloadDemo open System open System.IO open System.Reflection [<EntryPoint>] let main argv = let assemblyPath = argv.[0] // Path to the .dll to watch let mutable lastWriteTime = DateTime.MinValue while true do let writeTime = if File.Exists assemblyPath then File.GetLastWriteTimeUtc assemblyPath else lastWriteTime if writeTime > lastWriteTime then lastWriteTime <- writeTime try printfn "Last write time: %O " lastWriteTime printfn "Waiting for the build to finish (this is a hack)... " Threading.Thread.Sleep 10000 // 10s is plenty long enough for the build to finish printfn "Loading assembly path from: %s " assemblyPath let assembly = Assembly.LoadFrom assemblyPath printfn "Got assembly: %O" (assembly.GetName ()) let foo : (Unit -> int) option = assembly.GetExportedTypes() |> Array.tryHead |> Option.bind (fun t -> t.GetMethod "foo" |> Option.ofObj) |> Option.map (fun m -> (fun () -> m.Invoke (null, Array.empty) :?> int)) match foo with | Some foo -> printfn "foo () = %O" (foo ()) | None -> printfn "foo not found" with exn -> printfn "%O" exn else () Threading.Thread.Sleep 1000 0 I then have a very simple library to be watched in another project like this: module HotReload let foo () = 123456 To test it, I launch the "watcher" program. It successfully loads and invokes foo. Then, I modify my library (e.g. to return a different number) and build it with dotnet build. The watcher detects the change, loads the assembly again and invokes foo, but it prints the number from before the change! Then, I modify the library again with a different number. It detects the change but crashes: ... Loading assembly path from: ../hot-reload-lib/bin/Debug/netstandard2.0/hot-reload-lib.dll System.BadImageFormatException: Could not load file or assembly '<Unknown>'. Index not found. (0x80131124) File name: '<Unknown>' at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly) at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath) at System.Reflection.Assembly.LoadFrom(String assemblyFile) ... What is going on here? dotnet --version 3.0.100 lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic
No output from non standard File Descriptor
I am having issues receiving data from the child process. To send a string to the child process you write to FD3 and it will output the result to FD4. The child process runs fine, and if it couldn't write to FD4, the process would not start correctly, so FD4 has to be available but I just don't know why no output is given. My initial thoughts is when sending the string (to FD3) it wasn't null byte terminating therefore not receiving the string correctly (and then not sending anything back on FD4) but I am sure I am doing it correctly. I have tested writing to FD4 within the child process manually and the parent receives the output. use nix::fcntl::FcntlArg::{F_SETFD}; use nix::fcntl::{fcntl, open, FdFlag, OFlag}; use nix::sys::socket::{socketpair, AddressFamily, SockFlag, SockType}; use nix::sys::stat::Mode; use nix::unistd::{close, dup2, execvp, fork, pipe, read, write, ForkResult}; use std::ffi::CString; use std::os::unix::io::RawFd; use std::process::abort; fn main() { let input_socket: (RawFd, RawFd) = create_socket(); let output_socket: (RawFd, RawFd) = create_socket(); match fork() { Ok(ForkResult::Parent { child, .. }) => { println!("Child PID: {}", child); close(input_socket.1).unwrap(); close(output_socket.1).unwrap(); let test = r#"{"id":0,"method":"Target.getTargets"}\0"#; write(input_socket.0, test.as_bytes()).expect("unable to write"); let mut buf = [0; 64]; read(output_socket.0, &mut buf).unwrap(); println!("BUFFER: {:#?}", std::str::from_utf8(&buf).unwrap()); } Ok(ForkResult::Child) => { setup_child(input_socket.1, output_socket.1); abort() } Err(err) => { println!("{}", err); abort()}, } } #[cfg(any( target_os = "android", target_os = "dragonfly", target_os = "emscripten", target_os = "freebsd", target_os = "linux", target_os = "netbsd", target_os = "openbsd" ))] fn create_socket() -> (RawFd, RawFd) { socketpair( AddressFamily::Unix, SockType::Stream, None, SockFlag::SOCK_CLOEXEC, ) .unwrap() } #[cfg(any(target_os = "ios", target_os = "macos"))] fn create_socket() -> (RawFd, RawFd) { let socket = socketpair( AddressFamily::Unix, SockType::Stream, None, SockFlag::empty(), ) .unwrap(); fcntl(socket.0, F_SETFD(FdFlag::FD_CLOEXEC)).unwrap(); fcntl(socket.1, F_SETFD(FdFlag::FD_CLOEXEC)).unwrap(); socket } fn setup_child(input: RawFd, output: RawFd) { let _input: RawFd = dup2(input, 3).unwrap(); let _output: RawFd = dup2(output, 4).unwrap(); let file = CString::new("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome").unwrap(); let arg1 = CString::new("--remote-debugging-pipe").unwrap(); let arg2 = CString::new("--enable-logging=stderr").unwrap(); let args = vec![arg1.as_c_str(),arg2.as_c_str()]; let _res = execvp(&file, &args).unwrap(); } This is the open FDs for the child process: lsof -p 14620 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME Chrome 14620 tom 0 PIPE 0xcafab653bf79be56 16384 ->0xfc41206e43b6bb9d Chrome 14620 tom 1 PIPE 0x71c3c6a8bffbf5ad 16384 ->0x587f975b5bfd4499 Chrome 14620 tom 2 PIPE 0x71c3c6a8bffbf5ad 16384 ->0x587f975b5bfd4499 Chrome 14620 tom 3u unix 0x36cd3ac44c49d68d 0t0 ->0x36cd3ac44c49f2ad Chrome 14620 tom 4u unix 0x36cd3ac44c49dc05 0t0 ->0x36cd3ac44c49f9b5 UPDATE I found the issue, for some reason the first argument when running the chrome process was being ignored?? So when I place any other argument as the first in the list chrome actually starts up the remote debugging. Weird issue! When I run the chrome process in terminal with --remote-debugging-pipe as the first argument it works fine too, so why does it happen when i use: execvp(&file, &args)
The problem is how I am running the process. execvp(&file, &args) requires the first arg to be the file location. Thanks to: source Changing this code makes everything run as expected: let args = vec![file.as_c_str(), arg1.as_c_str(),arg2.as_c_str()];
Ecto.Repo receives a struct that does not implement Access behaviour
i have a problem with an Ecto Repo and a schema in one of my tests. The schema is the following: defmodule Elixirserver.Transactions.Bank do #behaviour Elixirserver.ContentDump use Ecto.Schema import Ecto.Changeset alias Elixirserver.Transactions.Account #attrs [:name, :code] schema "banks" do field(:name, :string) field(:code, :string) has_many(:account, Account) timestamps() end #doc false def changeset(bank, attrs \\ []) do bank |> cast(attrs, #attrs) |> validate_required(#attrs) end def to_json(bank) do %{ id: bank.id, name: bank.name, code: bank.code, type: "BANK" } end end When i try to execute a test i obtain the following: (UndefinedFunctionError) function Elixirserver.Transactions.Bank.fetch/2 is undefined (Elixirserver.Transactions.Bank does not implement the Access behaviour) The test is this: def create(conn, %{"bank" => bank_params}) do with {:ok, %Bank{} = bank} <- Transactions.create_bank(bank_params) do conn |> put_status(:created) |> put_resp_header("location", bank_path(conn, :show, bank)) |> render("show.json", id: bank["id"]) end end Now, apparently this is because the Access behaviour is not implemented. Do i have to provide it explicitly ? I am using ExMachina to generate fixtures, and i generated the resources with mix phx.gen.json.
bank["id"] is most probably the problem. Structs don't implement the access interface, you should use the dot so this should work: bank.id. Details can be found here.
Ejabberd: error in simple module to handle offline messages
I have an Ejabberd 17.01 installation where I need to push a notification in case a recipient is offline. This seems the be a common task and solutions using a customized Ejabberd module can be found everywhere. However, I just don't get it running. First, here's me script: -module(mod_offline_push). -behaviour(gen_mod). -export([start/2, stop/1]). -export([push_message/3]). -include("ejabberd.hrl"). -include("logger.hrl"). -include("jlib.hrl"). start(Host, _Opts) -> ?INFO_MSG("mod_offline_push loading", []), ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_message, 10), ok. stop(Host) -> ?INFO_MSG("mod_offline_push stopping", []), ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_message, 10), ok. push_message(From, To, Packet) -> ?INFO_MSG("mod_offline_push -> push_message", [To]), Type = fxml:get_tag_attr_s(<<"type">>, Packet), % Supposedly since 16.04 %Type = xml:get_tag_attr_s(<<"type">>, Packet), % Supposedly since 13.XX %Type = xml:get_tag_attr_s("type", Packet), %Type = xml:get_tag_attr_s(list_to_binary("type"), Packet), ?INFO_MSG("mod_offline_push -> push_message", []), ok. The problem is the line Type = ... line in method push_message; without that line the last info message is logged (so the hook definitely works). When browsing online, I can find all kinds of function calls to extract elements from Packet. As far as I understand it changed over time with new releases. But it's not good, all variants lead in some kind of error. The current way returns: 2017-01-25 20:38:08.701 [error] <0.21678.0>#ejabberd_hooks:run1:332 {function_clause,[{fxml,get_tag_attr_s,[<<"type">>,{message,<<>>,normal,<<>>,{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>},[],[{text,<<>>,<<"sfsdfsdf">>}],undefined,[],#{}}],[{file,"src/fxml.erl"},{line,169}]},{mod_offline_push,push_message,3,[{file,"mod_offline_push.erl"},{line,33}]},{ejabberd_hooks,safe_apply,3,[{file,"src/ejabberd_hooks.erl"},{line,382}]},{ejabberd_hooks,run1,3,[{file,"src/ejabberd_hooks.erl"},{line,329}]},{ejabberd_sm,route,3,[{file,"src/ejabberd_sm.erl"},{line,126}]},{ejabberd_local,route,3,[{file,"src/ejabberd_local.erl"},{line,110}]},{ejabberd_router,route,3,[{file,"src/ejabberd_router.erl"},{line,87}]},{ejabberd_c2s,check_privacy_route,5,[{file,"src/ejabberd_c2s.erl"},{line,1886}]}]} running hook: {offline_message_hook,[{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>},{message,<<>>,normal,<<>>,{jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>},{jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>},[],[{text,<<>>,<<"sfsdfsdf">>}],undefined,[],#{}}]} I'm new Ejabberd and Erlang, so I cannot really interpret the error, but the Line 33 as mentioned in {mod_offline_push,push_message,3,[{file,"mod_offline_push.erl"}, {line,33}]} is definitely the line calling get_tag_attr_s. UPDATE 2017/01/27: Since this cost me a lot of headache -- and I'm still not perfectly happy -- I post here my current working module in the hopes it might help others. My setup is Ejabberd 17.01 running on Ubuntu 16.04. Most stuff I tried and failed with seem to for older versions of Ejabberd: -module(mod_fcm_fork). -behaviour(gen_mod). %% public methods for this module -export([start/2, stop/1]). -export([push_notification/3]). %% included for writing to ejabberd log file -include("ejabberd.hrl"). -include("logger.hrl"). -include("xmpp_codec.hrl"). %% Copied this record definition from jlib.hrl %% Including "xmpp_codec.hrl" and "jlib.hrl" resulted in errors ("XYZ already defined") -record(jid, {user = <<"">> :: binary(), server = <<"">> :: binary(), resource = <<"">> :: binary(), luser = <<"">> :: binary(), lserver = <<"">> :: binary(), lresource = <<"">> :: binary()}). start(Host, _Opts) -> ?INFO_MSG("mod_fcm_fork loading", []), % Providing the most basic API to the clients and servers that are part of the Inets application inets:start(), % Add hook to handle message to user who are offline ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_notification, 10), ok. stop(Host) -> ?INFO_MSG("mod_fcm_fork stopping", []), ejabberd_hooks:add(offline_message_hook, Host, ?MODULE, push_notification, 10), ok. push_notification(From, To, Packet) -> % Generate JID of sender and receiver FromJid = lists:concat([binary_to_list(From#jid.user), "#", binary_to_list(From#jid.server), "/", binary_to_list(From#jid.resource)]), ToJid = lists:concat([binary_to_list(To#jid.user), "#", binary_to_list(To#jid.server), "/", binary_to_list(To#jid.resource)]), % Get message body MessageBody = Packet#message.body, % Check of MessageBody is not empty case MessageBody/=[] of true -> % Get first element (no idea when this list can have more elements) [First | _ ] = MessageBody, % Get message data and convert to string MessageBodyText = binary_to_list(First#text.data), send_post_request(FromJid, ToJid, MessageBodyText); false -> ?INFO_MSG("mod_fcm_fork -> push_notification: MessageBody is empty",[]) end, ok. send_post_request(FromJid, ToJid, MessageBodyText) -> %?INFO_MSG("mod_fcm_fork -> send_post_request -> MessageBodyText = ~p", [Demo]), Method = post, PostURL = gen_mod:get_module_opt(global, ?MODULE, post_url,fun(X) -> X end, all), % Add data as query string. Not nice, query body would be preferable % Problem: message body itself can be in a JSON string, and I couldn't figure out the correct encoding. URL = lists:concat([binary_to_list(PostURL), "?", "fromjid=", FromJid,"&tojid=", ToJid,"&body=", edoc_lib:escape_uri(MessageBodyText)]), Header = [], ContentType = "application/json", Body = [], ?INFO_MSG("mod_fcm_fork -> send_post_request -> URL = ~p", [URL]), % ADD SSL CONFIG BELOW! %HTTPOptions = [{ssl,[{versions, ['tlsv1.2']}]}], HTTPOptions = [], Options = [], httpc:request(Method, {URL, Header, ContentType, Body}, HTTPOptions, Options), ok.
Actually it fails with second arg Packet you pass to fxml:get_tag_attr_s in push_message function {message,<<>>,normal,<<>>, {jid,<<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>, <<"homer">>,<<"xxx.xxx.xxx.xxx">>,<<"conference">>}, {jid,<<"carl">>,<<"xxx.xxx.xxx.xxx">>,<<>>,<<"carl">>, <<"xxx.xxx.xxx.xxx">>,<<>>}, [], [{text,<<>>,<<"sfsdfsdf">>}], undefined,[],#{}} because it is not xmlel Looks like it is record "message" defined in tools/xmpp_codec.hrl with <<>> id and type 'normal' xmpp_codec.hrl -record(message, {id :: binary(), type = normal :: 'chat' | 'error' | 'groupchat' | 'headline' | 'normal', lang :: binary(), from :: any(), to :: any(), subject = [] :: [#text{}], body = [] :: [#text{}], thread :: binary(), error :: #error{}, sub_els = [] :: [any()]}). Include this file and use just Type = Packet#message.type or, if you expect binary value Type = erlang:atom_to_binary(Packet#message.type, utf8)
The newest way to do that seems to be with xmpp:get_type/1: Type = xmpp:get_type(Packet), It returns an atom, in this case normal.
Error attempting to decode with wreq
I'm trying really hard to understand how to use lenses and wreq and its turning out to really slow me down. The error seems to be claiming there's some mismatched type here. I'm not sure exactly how to handle that though. I'm still fairly new to haskell and these lenses are pretty confusing. However, wreq seems to be cleaner, which is why I chose to use it. Can anyone help me understand what the error is, and how to fix it? I seem to run into alot of these type errors. I am aware that Maybe TestInfo won't be returned by my code at the moment. That's ok. That error know how to handle. This error however, I don't. Here is my code: Module TestInformation: {-# LANGUAGE OverloadedStrings #-} module TestInformation where import Auth import Network.Wreq import Control.Lens import Data.Aeson import Data.Aeson.Lens (_String) type TestNumber = String data TestInfo = TestInfo { TestId :: Int, TestName :: String, } instance FromJSON TestInfo getTestInfo :: Key -> TestNumber -> Maybe TestInfo getTestInfo key test = decode (res ^. responseBody . _String) where opts = defaults & auth ?~ oauth2Bearer key res = getWith opts ("http://testsite.com/v1/tests/" ++ test) Module Auth: module Auth where import qualified Data.ByteString as B type Key = B.ByteString The error: GHCi, version 7.10.1: http://www.haskell.org/ghc/ :? for help [1 of 2] Compiling Auth ( Auth.hs, interpreted ) [2 of 2] Compiling TestInformation ( TestInformation.hs, interpreted ) TestInformation.hs:36:18: Couldn't match type ‘Response body10’ with ‘IO (Response Data.ByteString.Lazy.Internal.ByteString)’ Expected type: (body10 -> Const Data.ByteString.Lazy.Internal.ByteString body10) -> IO (Response Data.ByteString.Lazy.Internal.ByteString) -> Const Data.ByteString.Lazy.Internal.ByteString (IO (Response Data.ByteString.Lazy.Internal.ByteString)) Actual type: (body10 -> Const Data.ByteString.Lazy.Internal.ByteString body10) -> Response body10 -> Const Data.ByteString.Lazy.Internal.ByteString (Response body10) In the first argument of ‘(.)’, namely ‘responseBody’ In the second argument of ‘(^.)’, namely ‘responseBody . _String’ TestInformation.hs:36:33: Couldn't match type ‘Data.ByteString.Lazy.Internal.ByteString’ with ‘Data.Text.Internal.Text’ Expected type: (Data.ByteString.Lazy.Internal.ByteString -> Const Data.ByteString.Lazy.Internal.ByteString Data.ByteString.Lazy.Internal.ByteString) -> body10 -> Const Data.ByteString.Lazy.Internal.ByteString body10 Actual type: (Data.Text.Internal.Text -> Const Data.ByteString.Lazy.Internal.ByteString Data.Text.Internal.Text) -> body10 -> Const Data.ByteString.Lazy.Internal.ByteString body10 In the second argument of ‘(.)’, namely ‘_String’ In the second argument of ‘(^.)’, namely ‘responseBody . _String’ Failed, modules loaded: Auth. Leaving GHCi.
This type checks for me: getTestInfo :: Key -> TestNumber -> IO (Maybe TestInfo) getTestInfo key test = do res <- getWith opts ("http://testsite.com/v1/tests/" ++ test) return $ decode (res ^. responseBody) where opts = defaults & auth ?~ oauth2Bearer key getWith is an IO action, so to get its return value you need to use the monadic binding operator <-. Full program: http://lpaste.net/133443 http://lpaste.net/133498