Last time I used ELK there was no easy way to do housekeeping (e.g: automatically expiring old logs from the index) Have you added such a feature, or are you planning to?
Have a look at curator. It's open source and built by elastic. But more importantly we will have index management feature in 7.0 which will simplify even more all that.

You talked about a limit of 10 shards for node, collecting as in the istance 90days of logs with a daily index it will be hard to remain under that limit (we usually collect hundreds giga of data daily). Or the limit is just regarding the shards in a sing
It was just the limit we had for the given hardware we had. But you can have a hot/warm/cold architecture with elasticsearch. On the warm nodes, you can call the shrink API and reduce the number of shards, call force merge and then you will probably to st

In your scenario with an Index per day, if traffic increases, is the suggested strategy to create a new Index with more shards and switch the alias?
Exactly. Even better have a look at the Rollover API which can do that for you (creating a new index based on the number of docs you have or after a given delay and switch the alias)

Could not be a good idea to use kafka to decouple the pipeline and send logs to different logstash to perform different enrichments?
Totally a good idea if you want to scale out logstash nowadays. May be in the future, logstash will more act as a cluster.

Is it possible to archive old logs offline and later use them for query purposes?
Yes. But to query them you will have first to restore them. Might be better to use the coming Rollup index feature in 7.0.

FaaS is a portable technology? Can I deploy my function on another FaaS and will it work in the same way?
The various platforms are similar but not identical, they have minor differences in parameters for example. Check out the Serverless Framework which aims to make your code deployable to multiple serverless platforms https://serverless.com

Can I run my function locally on a dockerized environment offline?
You can run Apache OpenWhisk locally, it's great for development. I've also seen developers use simple scripts to wrap their functions for rapid local development.

Does the storer call the next function or the main function is calling it after storer function is done?
There is a sequence called 'qhandler' that links storer and notifier. My code calls the sequence and this makes storer run and the output of that becomes the input of notifier. In case seeing the code would help, its here https://github.com/ibm-watson-dat

Is there a pwa boilerplate with pouchdb, service workers and cloudant authentication?
Try these sample apps that were built by me and my team https://github.com/ibm-watson-data-lab/shopping list - and if you need more in terms of authentication or structure, take a look at hoodie.io

Often source data or destinationation db are under vpn or under private networks not easily accessible by cloud services. Are the cloud serverless tecnologies ready to manage this kind scenario and what solution they propose? Thank you and compliments for
For that kind of scenario, you can either put your serverless platform within the secure network, or a hybrid cloud solution may help, talk to your cloud provider

Quante applicazioni e stato necessario riscrivere, sino stai conteggiati i costi?
Abbiamo riscritto tutti i main website (Vanity Fair, Vogue, GQ, Glamour, Wired, La Cucina Italiana), il nostro DAM per essere AWS compliant e il PhotoVogue, applicativo verticale di Vogue dedicato ai fotografi di moda che vogliono caricare il proprio portfolio

Have you explored EKS or Fargate?
No, ma stiamo lavorando a un primo POC con sia con Fargate che con EKS, questo per permetterci di capire non solo l’impatto economico nell’utilizzo dei cluster gestiti, ma anche l’impatto a livello di performance o quanto dobbiamo modificare i nostri container per ottimizzare al meglio l’utilizzo delle risorse.

Without fargate, how is it possible to scale the services? we are limited by the number of physical instances, right?
We can scale both instances and container services. Thanks to different autoscaling groups we can use instaces metrics to add and remove ec2s and thanks to task autoscaling we can also scale containers inside multiple instances using application metrics...

What's the best approach to use existing binaries (of thirty part sw) in serverless env
The best approaches depend on the situation: A) You can create a static binary using Docker Lambda or a EC2 Instance with the same Linux. Then you can deploy that static binary either with your code or to upload it to S3 and download it upon your Lambda execution. B) Some binaries you can not run with Lambda (f.e. its a lib for Windows or its using things that are not exposed to you in Docker Lambda). In that case, the best way is to use AWS Fargate or eventually put it on your server and just invoke it with your Lambda.

The biggest challenge splitting a monolith usually is how to make splitted components, functions here, intercommunicate or, better, split them in a way that they intercommunicate as little as possible identifying the domain boundaries. Advices?
Splitting the monolith is the biggest challenge and it is a very big topic, spreading through many layers. We've explained in the lecture just the recommended approach to split your monolith into functions (with a focus on business processes) and how to think about migration. But there isn't a silver bullet for every use case. So there are some general rules / advices: 1. The main focus should actually be on the business process (in a certain domain of course) and thinking about the business process steps. We can talk a lot about the shifting focus from domains to business processes with serverless. Ping me on Twitter, thats a great topic to discuss together. 2. Serverless favorizes splitting into functions not domain microservices / components. As each function represents a 'task' / 'step' in the process. That seems hard from looking at my monolith but is actually enjoyable when you approach from the business perspective. 3. The communication between functions / services is almost free - your are getting charged only for your functions / services execution. The triggers for functions, connections between your serverless provider services and the functions are free. So the communication cost is not a problem. Latency is the only potential issue you should focus on. This topic is more for a full-on blogpost than a simple reply. Hope you reach me on Twitter or on Twitch.

There has been a lot of discussion about machine learning systems possibly sharing the prejudices of their creators. Do you think that can be problematic?
This is related to the creation of training datasets. Since data labelling is still done by humans, in some specific use cases there could be a human bias reflected in the ML model. It is a very interesting topic and there is active research in progress.

What about AutoML performance for DNA and illness predictions?
Currently only AutoML Vision has been released and it is tailored for image classification. New versions will come for other use cases but we don't have any benchmark on DNA/illness prediction at the moment.

What options does the advanced training method allow you to choose?
More training time and hyperparameter tuning.

Ehat kind of data and what features in the data can AutoML handle? (Besides images computrer visiion classification)
AutoML Vision can handle only images. New AutoML versions are coming soon and will handle new data types.

Is it possible to design or tune manually the NN structure in AutoML?
No, the purpose of AutoML is to automate these steps and provide better results than those achieved by humans.

The project is in alpha. When is planned a new release?
The exact date is under NDA.

Does out ML apply also model creation for TF object detection API
Not in the current stage but new features are added constantly.

Una volta creato un modello con AutoML è possibile esportarlo (il modello in sè e/o gli hyperparameter)?
Il modello viene automaticamente esportato su Cloud ML Engine per il serving ma non è al momento ulteriormente ispezionabile/esportabile.

Quale può essere una dimensione ragionevole minima di dataset per avere un proof of concept con AutoML?
Minimo un centinaio di sample per ogni label. Più aumenta la variabilità delle immagini da riconoscere, più è consigliabile un dataset ampio. Mediamente alcune migliaia di samples producono ottimi risultati.