French Opensource Advocate and Python Pragmatic Programmer (P³)
58 stories
·
0 followers

Some notes on fault tolerant storage systems

1 Share

If you care about how fault tolerant your storage is, you might find these articles and papers interesting. They have formed how I think of when designing a storage system.

Several of these research papers are based on data collected from hundred thousands or millions of disk, and their findings are eye opening. The short story is simply do not implicitly trust RAID or redundant storage systems. Details matter. And unfortunately there are few options on Linux addressing all the identified issues. Both ZFS and Btrfs are doing a fairly good job, but have legal and practical issues on their own. I wonder how cluster file systems like Ceph do in this regard. After all, there is an old saying, you know you have a distributed system when the crash of a compyter you have never heard of stops you from getting any work done. The same holds true if fault tolerance do not work.

Just remember, in the end, it do not matter how redundant, or how fault tolerant your storage is, if you do not continuously monitor its status to detect and replace failed disks.

Read the whole story
encolpe
46 days ago
reply
Lyon, France
Share this story
Delete

Python 3: Variable Annotations

1 Share

Python added syntax for what something called Variable Annotation in the version 3.6. Variable Annotation is basically an enhancement of type hinting, which was introduced in Python 3.5. The full explanation behind Variable Annotation is explained in PEP 526. In this article, we will have a quick refresher on type hinting and then introduce the new Variable Annotation syntax.

What’s Type Hinting?

Type hinting in Python is basically declaring that the parameters in your functions and methods have a certain type. Python does not enforce the type “hints”, but you can use tools like mypy to enforce the type hints in much the same way that C++ enforces type declarations at run time. Let’s look at a normal function with no type hints added:

def add(a, b):
    return a + b
 
if __name__ == '__main__':
    add(2, 4)

Here we create an add() function that takes two parameters. It adds the two parameters and returns the result. What we don’t know is what we need to pass to the function. We could pass it integers, floats, lists, or strings and it would most likely work. But would it work the way we expected it to? Let’s add some type hinting to our code:

def add(a: int, b: int) -> int:
    return a + b
 
if __name__ == '__main__':
    print(add(2, 4))

Here we change the definition of the function to utilize type hinting. You will notice that now the parameters are marked with what type they should be:

  • a: int
  • b: int

We also hint at the return value, which is what the “-> int” is. That means we expect an integer as our return value. If you try calling the add() function with a couple of strings or a float and an integer though, you won’t see an error. As I said, Python just allows you to hint at what the type of the parameters should be, but it does not enforce it.

Let’s update the code to the following:

def add(a: int, b: int) -> int:
    return a + b
 
if __name__ == '__main__':
    print(add(5.0, 4))

If you run this, you will see it executes just fine. Now let’s install mypy using pip:

pip install mypy

Now that we have mypy, we can use it to determine if we are using our function correctly. Open up a terminal and navigate to the folder that you save the above script to. Then execute the following command:

mypy hints.py

When I ran this command, I received the following output:

hints.py:5: error: Argument 1 to "add" has incompatible type "float"; expected "int"

As you can see, mypy found a problem with our code. We are passing in a float for the first argument instead of an int. You can use mypy on a continuous integration server that can check your code for these kinds of issues before committing pushing your submission to your branch or run it locally before committing your code at all.


Variable Annotation

Let’s say you want to not just annotate function parameters but also regular variables. In Python 3.5, you couldn’t do this using the same kind of syntax as you would for function parameters as it would raise a SyntaxError. Instead you would need to use comments, but now that 3.6 is out, we can use the new syntax! Let’s look at an example:

from typing import List
 
def odd_numbers(numbers: List) -> List:
    odd: List[int] = []
    for number in numbers:
        if number % 2:
            odd.append(number)
 
    return odd
 
if __name__ == '__main__':
    numbers = list(range(10))
    print(odd_numbers(numbers))

Here he specify that the variable, odd, should be a list of integers. If you run mypy against this script, you will receive no output as we are doing everything correctly. Let’s try changing the code to add something other than an integer though!

from typing import List
 
def odd_numbers(numbers: List) -> List:
    odd: List[int] = []
    for number in numbers:
        if number % 2:
            odd.append(number)
 
    odd.append('foo')
 
    return odd
 
if __name__ == '__main__':
    numbers = list(range(10))
    print(odd_numbers(numbers))

Here we add a new line that appends a string to the list of integers. Now if we run mypy against this version of the the code we should see the following:

hints2.py:9: error: Argument 1 to "append" of "list" has incompatible type "str"; expected "int"

Just to reiterate, in Python 3.5 you could do variable annotation, but you had to put that annotation in a comment:

# Python 3.6
odd: List[int] = []
 
# Python 3.5
odd = [] # type: List[int]

Note that if you change the code to use the Python 3.5 variation of variable annotation syntax, mypy will still flag the error correctly. You have to specify the “type:” after the pound sign though. If you remove that, then it’s no longer variable annotation. Basically all PEP 526 added was making the syntax more uniform across the language.


Wrapping Up

At this point you should have enough information to start doing variable annotation in your own code whether you’re using Python 3.5 or 3.6 I think it’s a neat concept and will be especially useful to those programmers that work with others that are more familiar with a static typed language.


Related Reading

Read the whole story
encolpe
46 days ago
reply
Lyon, France
Share this story
Delete

Comment apprendre plus rapidement ?

2 Shares

Ce n'est pas toujours facile d'apprendre de nouvelles choses, surtout quand on est toujours très occupé ou sans cesse dérangé. Heureusement, il existe quelques techniques simples mises en lumière par ASAP Science qui permettent de fixer une information dans votre mémoire de manière plus efficace et rapide.

En ce qui me concerne, je sais que quand j'étais étudiant, ce qui fonctionnait pour moi, c'était de prendre des notes, mais aussi de les relire le soir avant de m'endormir. Et pour le bac, comme c'était trop le bordel de reprendre tous mes cours de l'année, j'ai révisé en m'aidant uniquement d'un "ABC du BAC tout-en-un" + un max d'exo pratiques. Ce qu'il faut essayer de déterminer surtout, c'est la méthode qui vous convient le mieux, car certaines personnes sont plus visuelles ou plus auditives que d'autres.

Pour compléter cette vidéo, je vous ai aussi dégoté une infographie sympa qui reprend un peu tout ça et qui y ajoute un peu d'exercice physique afin de répondre à l'adage "Un esprit sain dans un corps sain".

Source

Cet article merveilleux et sans aucun égal intitulé : Comment apprendre plus rapidement ? ; a été publié sur Korben, le seul site qui t'aime plus fort que tes parents.

Read the whole story
encolpe
76 days ago
reply
Lyon, France
Share this story
Delete

TLS Authentication on Freenode and OFTC

1 Share

In order to easily authenticate with IRC networks such as OFTC and Freenode, it is possible to use client TLS certificates (also known as SSL certificates). In fact, it turns out that it's very easy to setup both on irssi and on znc.

Generate your TLS certificate

On a machine with good entropy, run the following command to create a keypair that will last for 10 years:

openssl req -nodes -newkey rsa:2048 -keyout user.pem -x509 -days 3650 -out user.pem -subj "/CN=<your nick>"

Then extract your key fingerprint using this command:

openssl x509 -sha1 -noout -fingerprint -in user.pem | sed -e 's/^.*=//;s/://g'

Share your fingerprints with NickServ

On each IRC network, do this:

/msg NickServ IDENTIFY Password1!
/msg NickServ CERT ADD <your fingerprint>

in order to add your fingerprint to the access control list.

Configure ZNC

To configure znc, start by putting the key in the right place:

cp user.pem ~/.znc/users/<your nick>/networks/oftc/moddata/cert/

and then enable the built-in cert plugin for each network in ~/.znc/configs/znc.conf:

<Network oftc>
    ...
            LoadModule = cert
    ...
</Network>
    <Network freenode>
    ...
            LoadModule = cert
    ...
</Network>

Configure irssi

For irssi, do the same thing but put the cert in ~/.irssi/user.pem and then change the OFTC entry in ~/.irssi/config to look like this:

{
  address = "irc.oftc.net";
  chatnet = "OFTC";
  port = "6697";
  use_tls = "yes";
  tls_cert = "~/.irssi/user.pem";
  tls_verify = "yes";
  autoconnect = "yes";
}

and the Freenode one to look like this:

{
  address = "chat.freenode.net";
  chatnet = "Freenode";
  port = "7000";
  use_tls = "yes";
  tls_cert = "~/.irssi/user.pem";
  tls_verify = "yes";
  autoconnect = "yes";
}

That's it. That's all you need to replace password authentication with a much stronger alternative.

Read the whole story
encolpe
99 days ago
reply
Lyon, France
Share this story
Delete

A laptop by KDE

1 Share

Earlier this year we announced a joint venture between KDE and Slimbook that we named the KDE Slimbook.

Last Akademy we had the opportunity to meet the Slimbook team and discuss its purpose and future. I’m quite happy about the discussions, here’s my feedback.

Why?

Before talking about future stuff, let us to set the context.

Personally, this has been a long-standing pet-peeve of mine. We create software solutions for virtual hardware specifications. We require our users to go the extra mile by replacing the manufacturer’s operating system, often with some annoyances because their hardware isn’t properly supported. This is not necessarily our fault, but most of us agree that we should improve this situation.

On the software side of things, while it’s our area of expertise, we are still far from being able to deliver the product we would like to give. Our ecosystem is still complex and our users have to take several decisions, some of dubious nature.

Hence, I started pushing for this project with some colleagues from KDE.

Where are we?

As discussed in the Akademy presentation, the project worked reasonably well. Devices were delivered and the KDE community does see value in the project.

We have reached a point where we understand the value of the initiative and understand the environment much better. We are happy with the results so far and we see room for improvement; both in the task we do to make a better device as well as in defining the product.

Future

I want to see the project moving forward and I will keep working towards better cohesion between hardware vendors and KDE. In the future, I would like to see the devices that we see available, especially when it comes to different use cases, and therefore form factors.

I don’t think it makes sense that we continue doing it just to do so. There is a good deal of hardware vendors that will offer devices with good setups. Most of them they will even offer you these laptops with KDE on them upon demand. If we put our effort in such projects, it needs to be because we are making something special.

Let’s not get ahead of ourselves. One step at a time. Let’s continue to provide good solutions by staying simple by default but powerful when needed.

Read the whole story
encolpe
129 days ago
reply
Lyon, France
Share this story
Delete

Mozilla Common Voice : vers une reconnaissance vocale libre

1 Share

La fondation Mozilla a discrètement lancé le projet Common Voice, qui a pour objectif de fournir une base de données de voix à destination des développeurs de logiciels de reconnaissance vocale. D'ici la fin de l'année, la base de données sera mise à disposition de la communauté sous licence CC-0, et pourra être utilisée par les développeurs pour entraîner leurs logiciels de reconnaissance vocale.

Il est actuellement possible de participer depuis le web, en lisant des phrases en anglais et en indiquant son accent, son âge et son genre pour enrichir la base de données. Des applications sont déjà en cours de développement pour les appareils mobiles et devraient être disponibles prochainement. Pour ceux qui ne parlent pas anglais, il est aussi possible de contribuer au projet en produisant des phrases pertinentes et/ou du code dans github.

Lire les commentaires

Read the whole story
encolpe
137 days ago
reply
Lyon, France
Share this story
Delete
Next Page of Stories